What is SWE-bench? Verified, Pro and how scores work
SWE-bench tests whether an AI agent can fix real GitHub issues. Learn how a task is scored, what Verified and Pro are, and why their scores differ.
SWE-bench is a benchmark that asks an AI system to fix real software issues. Each task gives the system a GitHub issue and a copy of the repository, and the system returns a code patch. The original paper (Jimenez et al., first submitted 2023-10-10, ICLR 2024) has 2,294 tasks from real issues and pull requests, and reported that its best model, Claude 2, solved 1.96% of them.
How a task is scored
The paper builds each task from a merged pull request in one of 12 popular Python repositories, in three stages. It collects about 90,000 pull requests, keeps those that resolve an issue and change test files, and keeps only instances with at least one test that fails before the change and passes after. Instances with install or runtime errors are dropped, which leaves 2,294.
The model sees the issue text and the codebase at the pull request’s base commit. A task counts as resolved only when the patch applies and two sets of tests pass:
- FAIL_TO_PASS tests, which check that the issue is fixed.
- PASS_TO_PASS tests, which check that existing behavior still works.
The score is the percentage of instances resolved. The official harness applies each patch inside Docker and runs the repository tests. It reports an empty patch or an error separately from a failed test.
The variants
The swebench.com leaderboards (read 2026-10-30) have tabs for Verified, Multimodal, Multilingual, Lite and Full, plus a Bash Only view. Full is the original benchmark (overview).
| Name | What it is | Source |
|---|---|---|
| Full | The original 2,294 tasks | Paper |
| Verified | 500 human-validated instances, made with OpenAI | swebench.com |
| Bash Only | A view of Verified that compares models with mini-SWE-agent in a minimal bash environment | swebench.com |
| Multilingual, Multimodal, Lite | Other tabs on the same site | swebench.com |
| Pro | 1,865 problems from 41 repositories, built by Scale AI | Paper, Scale leaderboard |
The swebench.com Bash Only page says its setup is not tuned to reach higher scores, and that release 1.x and 2.x results may not be directly comparable because 2.x uses tool calling.
Verified
Verified exists because some original tasks were unclear or had tests that did not fit the issue. Annotators kept 500 instances whose descriptions were clear, tests correct and fix possible from the information given (swebench.com).
Since 2025-11-18, Verified and Multilingual accept only submissions that link an arXiv preprint or technical report with an author from an academic institution or established research lab (submission rules, read 2026-10-30).
SWE-bench Pro
Scale AI’s SWE-Bench Pro paper (arXiv v1 2025-09-21, v2 2025-11-14) has 1,865 problems from 41 actively maintained repositories. A public set uses 11 repositories, a held-out set 12 and a commercial set 18 proprietary ones. Its tasks may take a professional engineer hours to days and often change several files. The authors call it a “contamination-resistant testbed”.
The paper’s first version reported GPT-5 as the top model at 23.3% pass@1 (arXiv v1). The second version says models stay below 45% pass@1, with Claude Sonnet 4.5 highest at 43.6% on the public set with SWE-Agent and no cost cap (Hugging Face paper page, read 2026-10-30). On the Scale leaderboard page (read 2026-10-30), Claude Opus 4.1 falls from 22.7% on the public set to 17.8% on the private set, and GPT-5 from 23.1% to 14.9%. Scale’s page lists higher scores for newer models, so check it for current numbers.
Reading a score
- Name the subset. A Verified percentage and a Pro percentage measure different tasks.
- Name the scaffold. Scale marks leaderboard rows that use the mini-swe-agent harness, and swebench.com separates Bash Only runs, so the same model can post different numbers with a different agent loop.
- Check the date. Leaderboards change, and this page records the day each figure was read.
- Treat a test-based pass as one signal. Resolved means the tests passed, not that the patch is how a maintainer would write it.
To measure an agent on your own code, build a test set from your own issues and run it as a regression eval. See what LLM evaluation is.
FAQ
What does SWE-bench Verified mean?
It is a human-validated subset of 500 SWE-bench instances, made with OpenAI. Annotators checked that each problem description was clear, the tests were correct and the task was solvable from the information given.
What is the difference between SWE-bench Verified and SWE-bench Pro?
Verified keeps 500 tasks from the original Python repositories. SWE-Bench Pro has 1,865 problems from 41 repositories, including private commercial ones, with tasks that often span several files. A percentage on one is not comparable with a percentage on the other.
What does a SWE-bench score mean?
It is the share of task instances resolved, meaning the submitted patch applies and the repository's tests pass. It depends on the model, the agent scaffold around it and the subset, so quote all three.