# Agent benchmarks explained: SWE-bench, OSWorld, WebArena, BrowseComp and more

> Agent benchmarks score AI agents on tasks like fixing code, using a computer or browsing the web. What each measures, how it is graded, and how to read a score.

Source: https://www.honeyhive.ai/learn/agent-benchmarks (updated 2026-11-08)

Agent benchmarks are tests that put an AI agent in an interactive environment, give it a goal and grade the result. They differ from question-answering benchmarks because the agent takes many steps, calls tools and changes state, and the grader checks that state or the final answer. This page compares the main ones by what they test, how they grade and what to watch for when you read a score.

## The benchmarks at a glance

| Benchmark | What the agent does | How it is graded | Size | Source |
|---|---|---|---|---|
| [SWE-bench](/learn/swe-bench) | Fixes a real GitHub issue by returning a patch | The repository's tests pass | 2,294 tasks in the original; [500 in Verified](https://www.swebench.com/verified.html) | [Paper](https://arxiv.org/abs/2310.06770) |
| [Terminal-Bench](/learn/terminal-bench) | Completes an end-to-end task in a terminal | Tests on the final container state | Changes by version | [Paper](https://arxiv.org/abs/2601.11868) |
| [tau-bench](/learn/tau-bench) | Handles a tool-using conversation with a simulated user | Final database state against a goal state | Retail and airline domains | [Paper](https://arxiv.org/abs/2406.12045) |
| OSWorld | Operates a real computer across apps | Execution-based scripts on the result | 369 tasks | [Paper](https://arxiv.org/abs/2404.07972) |
| WebArena | Completes goals on self-hosted websites | Answer matching or checks on site state | 812 tasks | [Paper](https://arxiv.org/abs/2307.13854) |
| BrowseComp | Searches the web for a hard-to-find fact | Match to a short reference answer | 1,266 questions | [Paper](https://arxiv.org/abs/2504.12516) |
| GAIA | Answers real-world questions using reasoning, browsing and tools | Match to a reference answer | 466 questions | [Paper](https://arxiv.org/abs/2311.12983) |
| AgentBench | Acts in 8 environments, from databases to games | Per-environment scores, weighted into one | 8 environments | [Paper](https://arxiv.org/abs/2308.03688) |

## Coding agents

[SWE-bench](/learn/swe-bench) and [Terminal-Bench](/learn/terminal-bench) each have their own page. The short version: SWE-bench grades a patch against a repository's tests, and Terminal-Bench grades the end state of a container.

SWE-bench Verified deserves a caution. OpenAI said in February 2026 that Verified had ["lost its value as a meaningful measure of AI coding ability"](https://the-decoder.com/openai-wants-to-retire-the-ai-coding-benchmark-that-everyone-has-been-competing-on/). According to The Decoder's report, OpenAI found that GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash Preview could reproduce some original fixes from memory, and it recommended SWE-bench Pro instead. A Verified score may reflect what a model has seen in training.

## Computer-use agents

[OSWorld](https://arxiv.org/abs/2404.07972) (Xie and colleagues, April 2024) has 369 tasks on real web and desktop apps, file operations and multi-app workflows, in an environment that runs Ubuntu, Windows and macOS. When it launched, humans completed over 72.36% of tasks and the best model 12.24%.

The project's [site](http://osworld-v1.xlang.ai/) announced OSWorld-Verified on July 28, 2025, with fixes for community-reported examples and AWS support that brings evaluation down to about an hour. It also lists OSWorld 2.0, dated June 26, 2026. Check which version a score uses before comparing it. See [computer use agents](/learn/computer-use-agents) for how these agents work.

## Web agents

[WebArena](https://arxiv.org/abs/2307.13854) (Zhou and colleagues, July 2023) runs agents against self-hosted copies of an e-commerce store, a Reddit-style forum, GitLab and a content-management admin. The [paper](https://arxiv.org/html/2307.13854) has 812 test tasks built from 241 templates. It grades on functional correctness, not on matching a reference action sequence. Information-seeking answers are checked by exact match, required content or a GPT-4 judge for equivalent meaning. Navigation and configuration tasks run programmatic checks on the resulting site state. Some tasks are unachievable on purpose, with an "N/A" answer, to test whether an agent avoids unfounded claims.

At publication, humans succeeded on 78.24% of tasks and the best GPT-4 agent on 14.41%.

## Browsing and research agents

[BrowseComp](https://arxiv.org/abs/2504.12516) (Wei and colleagues at OpenAI, April 2025) measures persistence in finding information that is hard to locate. Trainers wrote each of the 1,266 questions backwards from a seed fact, so the answer is short and easy to check but hard to find. The [paper](https://arxiv.org/html/2504.12516) says each question had to defeat GPT-4o, o1 and an early Deep Research model, and could not be solved by a second person in ten minutes.

The paper's baselines show how hard it was then: GPT-4o 0.6%, GPT-4o with browsing 1.9%, o1 9.9% and Deep Research 51.5%. Human trainers gave up on 70.8% of the questions they attempted after two hours. The authors note it does not test long-form answers or ambiguous queries.

[GAIA](https://arxiv.org/abs/2311.12983) takes a broader view of an assistant: 466 real-world questions that need reasoning, multimodal handling, browsing and tool use, with answers to 300 withheld for a public leaderboard. Its paper reported 92% for humans and 15% for GPT-4 with plugins. For the levels and scoring, see [the GAIA benchmark](/learn/gaia-benchmark).

## Multi-environment benchmarks

[AgentBench](https://arxiv.org/abs/2308.03688) (Liu and colleagues, ICLR 2024) has [8 environments](https://arxiv.org/html/2308.03688): operating system, database and knowledge graph tasks, three game-style tasks (a card game, lateral thinking puzzles and household tasks) and two web tasks (shopping and browsing). Its overall score weights each task by the reciprocal of the average score across models, so a task where models score high, such as web shopping, does not dominate. The authors found strong commercial models and weaker open models of 70B parameters or fewer, and named long-term reasoning, decision-making and instruction following as the main obstacles.

## Reading a score

- Name the version and subset. SWE-bench Verified, OSWorld-Verified and [Terminal-Bench](/learn/terminal-bench) 2.0 or [4.0](https://www.tbench.ai/news/terminal-bench-4-0) are different tests from their predecessors.
- Name the scaffold and the model. A score is a pair, and the same model can move a lot with a different agent loop.
- Look at the grader. Exact-match answers, test suites and LLM judges fail in different ways. WebArena uses all three kinds, and the numbers inherit their quirks.
- Treat early numbers as snapshots. Every benchmark above started with a low top score that later models raised, so a figure from a paper says little about today. Read current numbers from each project's official leaderboard and note the date.
- Expect contamination and fixes. Benchmarks get patched, versioned and retired, as the SWE-bench Verified case shows.

## Using benchmarks for your own agent

Public benchmarks help you pick a model or scaffold to try. They do not tell you whether your agent handles your users' tasks. For that, build a test set from your own traces and failures and run it on every change. See [AI agent evaluation](/learn/ai-agent-evaluation), [testing AI agents](/learn/testing-ai-agents) and [agent evaluation metrics](/learn/agent-evaluation-metrics).

## FAQ

### What are agent benchmarks?

Tests that give an AI agent a goal in an interactive setting, such as a code repository, a terminal, a desktop or a website, and grade the outcome. [AgentBench](https://arxiv.org/abs/2308.03688), first posted in August 2023, was an early multi-environment example with 8 environments.

### Which agent benchmark should I look at for coding agents?

[SWE-bench](/learn/swe-bench) for fixing repository issues and [Terminal-Bench](/learn/terminal-bench) for command-line tasks. OpenAI said in February 2026 that it recommends SWE-bench Pro over Verified, as [The Decoder reported](https://the-decoder.com/openai-wants-to-retire-the-ai-coding-benchmark-that-everyone-has-been-competing-on/).

### What does BrowseComp measure?

How persistently a browsing agent can find a hard-to-locate fact. The [paper](https://arxiv.org/abs/2504.12516) has 1,266 questions with short answers that are hard to find but easy to verify.

### Do benchmark scores predict how my agent will perform?

Only loosely. A public benchmark tests a fixed task set with a fixed scaffold, so use it to shortlist models and then run your own tests on your own tasks.