# What is Terminal-Bench? Versions 2.0 to 4.0 and how scores work

> Terminal-Bench tests whether AI agents can finish real command-line tasks, scored by tests on the final container state, across versions 2.0, 2.1, 3.0 and 4.0.

Source: https://www.honeyhive.ai/learn/terminal-bench (updated 2026-11-06)

Terminal-Bench is a benchmark that gives an AI agent a task in a real terminal, such as compiling code, training a model or setting up a server, and checks the result with tests. It is [hosted](https://www.tbench.ai/) by Stanford and the Laude Institute. The [paper](https://arxiv.org/abs/2601.11868) (Merrill, Shaw, Carlini and 82 others, January 2026) describes version 2.0, and the project has shipped three more versions since.

## How a task works

The [GitHub project](https://github.com/laude-institute/terminal-bench) has two parts: a dataset of tasks and a harness that connects a language model to a sandboxed terminal. Each task has an English instruction, a test script and a reference ("oracle") solution. Each runs in its own Docker container.

Scoring is all or nothing per task. The paper says the tests check the final container state and ["do not test the agent's commands or console output"](https://arxiv.org/html/2601.11868). An agent can reach the result by any route that leaves the system in the right state. The score is the share of tasks resolved, and a leaderboard row is a pair of an agent (the scaffold) and a model. Compare [SWE-bench](/learn/swe-bench), which tests code patches against a repository's tests, and [tau-bench](/learn/tau-bench), which tests tool-using conversations.

## Which version is it

The [official news page](https://www.tbench.ai/news) lists four task sets in less than a year. The site now describes version 4.0.

| Version | Date | What it is | Source |
|---|---|---|---|
| 2.0 | [November 2025](https://www.tbench.ai/news) | 89 tasks; paper released January 2026 | [Paper](https://arxiv.org/abs/2601.11868) |
| 2.1 | [May 2026](https://www.tbench.ai/news) | The same 89 tasks with 28 fixed | [2.1 post](https://www.tbench.ai/news/terminal-bench-2-1) |
| 3.0 | [July 2026](https://www.tbench.ai/news) | 74 new, harder tasks across 7 domains | [3.0 post](https://www.tbench.ai/news/terminal-bench-3-0) |
| 4.0 | [August 2026](https://www.tbench.ai/news) | 3.0 with resources recalibrated, 8 tasks removed, 19 fixed | [4.0 post](https://www.tbench.ai/news/terminal-bench-4-0) |

### 2.0

The paper reports that frontier models and agents scored below 65%. It ran each agent and model pair at least five times, for 32,155 trials. Codex CLI with GPT-5.2 led at 62.9% (plus or minus 3.0), ahead of Terminus 2 with Claude Opus 4.5 at 58% and Terminus 2 with Gemini 3 Pro at 57%. Terminus 2 is the project's own agent, built as a [neutral testbed](https://arxiv.org/html/2601.11868) for comparing models.

[Snorkel](https://www.snorkel.ai/blog/terminal-bench-2-0/), a top outside contributor, counts 16 categories, with software engineering the largest at 26 tasks, and a mix of 4 easy, 55 medium and 30 hard tasks.

### 2.1

The [2.1 post](https://www.tbench.ai/news/terminal-bench-2-1) fixes 28 of the 89 tasks. Nine had external dependencies that changed after the benchmark was built, even with pinned Docker images. Eight had resource budgets too small for at least one valid approach. Others had instructions that did not match their tests: in `query-optimize`, the tests expected Spark SQL output while the instructions asked for PostgreSQL. Most agent and model pairs improved, and the largest gain was 12.1 points (58.0% to 70.1%) for Claude Code with Opus 4.6.

### 3.0

The [3.0 post](https://www.tbench.ai/news/terminal-bench-3-0) says many earlier tasks had become saturated. The first release has 74 tasks across 7 domains, with longer-running tasks, multi-container networks and GPUs. It separates the agent container from the verifier, so results can be re-graded when a verifier changes. The post reports the best models near 34%, with GPT-5.6 Sol in Codex at 34.4% and Fable 5 in Claude Code at 33.8%. It adds that Fable 5 and Opus 4.8 differ by 4.9 points on 2.1 and 12.7 on 3.0, so the harder set separates top models better.

### 4.0

The [4.0 post](https://www.tbench.ai/news/terminal-bench-4-0) sets a flat 8-hour agent timeout and calibrates time, CPU and memory with the method in Anthropic's [infrastructure noise write-up](https://www.anthropic.com/research/infrastructure-noise). It removes 8 tasks (2 each for saturation, refusals, public solutions and quality or platform issues) and fixes 19. A task counts as saturated when every model class in the latest generation solves it 5 out of 5 times.

The authors call it 4.0 rather than 3.1 because resource changes and removals are breaking and need trials to be re-run. They now treat Terminal-Bench as a continuous benchmark with semantic versioning, and say 4.1 will focus on verifier improvements such as tamper-resistant verifiers.

## Reading a score

- Name the version. A 2.0 score and a 4.0 score are different tests, and the project says re-running is required across breaking versions.
- Name the agent and the model. Leaderboard rows pair them, so the same model moves with a different scaffold.
- Read the confidence interval. Snorkel found the top 10 of the 2.0 leaderboard (77.3% to 84.7%) have overlapping 95% confidence intervals, and the [official site](https://www.tbench.ai/) reports 95% intervals for 4.0.
- Check the date. Leaderboards move quickly, and the scores above come from each version's release post, current as of November 2026.
- Know the integrity rules. In an [April 2026 update](https://www.tbench.ai/news/leaderboard-integrity-update), the project required trajectories for all passing trials, set reward hacking to zero for the trial and said cheating gets a submission taken down.

## Why agents fail

The paper's error analysis sorts trajectory failures into Execution, Coherence and Verification, and Execution errors dominate for Opus 4.5 and GPT-5.2. At the command level, the most common failure (24.1% of failures) was calling an executable that was not installed or not on the PATH. The 4.0 post says remaining errors are mostly model refusals and output-token limits.

## Using it on your own agents

Terminal-Bench measures end state, so it tells you whether an agent finished, not how it got there. To see the path, read the [traces](/learn/agent-tracing), and to turn a failure into a regression check, see [LLM evaluation metrics](/learn/llm-evaluation-metrics) and [how to test AI agents](/learn/testing-ai-agents). Run the benchmark itself with Harbor and Docker, using the command in the [4.0 post](https://www.tbench.ai/news/terminal-bench-4-0).

## FAQ

### What is Terminal-Bench?

A benchmark of hard, end-to-end tasks in a terminal, such as compiling code, training models and setting up servers. The [GitHub project](https://github.com/laude-institute/terminal-bench) pairs a task dataset with a harness that connects a model to a sandboxed terminal.

### How is a Terminal-Bench task scored?

A task passes when its tests succeed against the final state of the container. The [paper](https://arxiv.org/abs/2601.11868) says the tests do not check the agent's commands or console output.

### Can I compare a Terminal-Bench 2.0 score with a 4.0 score?

No. The [4.0 release post](https://www.tbench.ai/news/terminal-bench-4-0) changes resource limits and removes tasks, which it calls breaking changes that require re-running trials.

### How do I run Terminal-Bench?

It runs on the Harbor framework. The 4.0 post gives the command `harbor run -d terminal-bench/terminal-bench@4.0.0`. You need Docker.