Agent benchmarks explained: SWE-bench, OSWorld, WebArena, BrowseComp and more

Agent benchmarks are tests that put an AI agent in an interactive environment, give it a goal and grade the result. They differ from question-answering benchmarks because the agent takes many steps, calls tools and changes state, and the grader checks that state or the final answer. This page compares the main ones by what they test, how they grade and what to watch for when you read a score.

The benchmarks at a glance

Benchmark	What the agent does	How it is graded	Size	Source
SWE-bench	Fixes a real GitHub issue by returning a patch	The repository’s tests pass	2,294 tasks in the original; 500 in Verified	Paper
Terminal-Bench	Completes an end-to-end task in a terminal	Tests on the final container state	Changes by version	Paper
tau-bench	Handles a tool-using conversation with a simulated user	Final database state against a goal state	Retail and airline domains	Paper
OSWorld	Operates a real computer across apps	Execution-based scripts on the result	369 tasks	Paper
WebArena	Completes goals on self-hosted websites	Answer matching or checks on site state	812 tasks	Paper
BrowseComp	Searches the web for a hard-to-find fact	Match to a short reference answer	1,266 questions	Paper
GAIA	Answers real-world questions using reasoning, browsing and tools	Match to a reference answer	466 questions	Paper
AgentBench	Acts in 8 environments, from databases to games	Per-environment scores, weighted into one	8 environments	Paper

Coding agents

SWE-bench and Terminal-Bench each have their own page. The short version: SWE-bench grades a patch against a repository’s tests, and Terminal-Bench grades the end state of a container.

SWE-bench Verified deserves a caution. OpenAI said in February 2026 that Verified had “lost its value as a meaningful measure of AI coding ability”. According to The Decoder’s report, OpenAI found that GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash Preview could reproduce some original fixes from memory, and it recommended SWE-bench Pro instead. A Verified score may reflect what a model has seen in training.

Computer-use agents

OSWorld (Xie and colleagues, April 2024) has 369 tasks on real web and desktop apps, file operations and multi-app workflows, in an environment that runs Ubuntu, Windows and macOS. When it launched, humans completed over 72.36% of tasks and the best model 12.24%.

The project’s site announced OSWorld-Verified on July 28, 2025, with fixes for community-reported examples and AWS support that brings evaluation down to about an hour. It also lists OSWorld 2.0, dated June 26, 2026. Check which version a score uses before comparing it. See computer use agents for how these agents work.

Web agents

WebArena (Zhou and colleagues, July 2023) runs agents against self-hosted copies of an e-commerce store, a Reddit-style forum, GitLab and a content-management admin. The paper has 812 test tasks built from 241 templates. It grades on functional correctness, not on matching a reference action sequence. Information-seeking answers are checked by exact match, required content or a GPT-4 judge for equivalent meaning. Navigation and configuration tasks run programmatic checks on the resulting site state. Some tasks are unachievable on purpose, with an “N/A” answer, to test whether an agent avoids unfounded claims.

At publication, humans succeeded on 78.24% of tasks and the best GPT-4 agent on 14.41%.

Browsing and research agents

BrowseComp (Wei and colleagues at OpenAI, April 2025) measures persistence in finding information that is hard to locate. Trainers wrote each of the 1,266 questions backwards from a seed fact, so the answer is short and easy to check but hard to find. The paper says each question had to defeat GPT-4o, o1 and an early Deep Research model, and could not be solved by a second person in ten minutes.

The paper’s baselines show how hard it was then: GPT-4o 0.6%, GPT-4o with browsing 1.9%, o1 9.9% and Deep Research 51.5%. Human trainers gave up on 70.8% of the questions they attempted after two hours. The authors note it does not test long-form answers or ambiguous queries.

GAIA takes a broader view of an assistant: 466 real-world questions that need reasoning, multimodal handling, browsing and tool use, with answers to 300 withheld for a public leaderboard. Its paper reported 92% for humans and 15% for GPT-4 with plugins. For the levels and scoring, see the GAIA benchmark.

Multi-environment benchmarks

AgentBench (Liu and colleagues, ICLR 2024) has 8 environments: operating system, database and knowledge graph tasks, three game-style tasks (a card game, lateral thinking puzzles and household tasks) and two web tasks (shopping and browsing). Its overall score weights each task by the reciprocal of the average score across models, so a task where models score high, such as web shopping, does not dominate. The authors found strong commercial models and weaker open models of 70B parameters or fewer, and named long-term reasoning, decision-making and instruction following as the main obstacles.

Reading a score

• Name the version and subset. SWE-bench Verified, OSWorld-Verified and Terminal-Bench 2.0 or 4.0 are different tests from their predecessors.
• Name the scaffold and the model. A score is a pair, and the same model can move a lot with a different agent loop.
• Look at the grader. Exact-match answers, test suites and LLM judges fail in different ways. WebArena uses all three kinds, and the numbers inherit their quirks.
• Treat early numbers as snapshots. Every benchmark above started with a low top score that later models raised, so a figure from a paper says little about today. Read current numbers from each project’s official leaderboard and note the date.
• Expect contamination and fixes. Benchmarks get patched, versioned and retired, as the SWE-bench Verified case shows.

Using benchmarks for your own agent

Public benchmarks help you pick a model or scaffold to try. They do not tell you whether your agent handles your users’ tasks. For that, build a test set from your own traces and failures and run it on every change. See AI agent evaluation, testing AI agents and agent evaluation metrics.

FAQ

What are agent benchmarks?

Tests that give an AI agent a goal in an interactive setting, such as a code repository, a terminal, a desktop or a website, and grade the outcome. AgentBench, first posted in August 2023, was an early multi-environment example with 8 environments.

Which agent benchmark should I look at for coding agents?

SWE-bench for fixing repository issues and Terminal-Bench for command-line tasks. OpenAI said in February 2026 that it recommends SWE-bench Pro over Verified, as The Decoder reported.

What does BrowseComp measure?

How persistently a browsing agent can find a hard-to-locate fact. The paper has 1,266 questions with short answers that are hard to find but easy to verify.

Do benchmark scores predict how my agent will perform?

Only loosely. A public benchmark tests a fixed task set with a fixed scaffold, so use it to shortlist models and then run your own tests on your own tasks.