What is OSWorld? The computer-use agent benchmark
OSWorld tests agents on 369 real computer tasks in virtual machines, scored by scripts that check the final state. It has a Verified update and a 2.0 successor.
OSWorld is a benchmark that puts an AI agent in front of a real computer and asks it to finish a task, such as reformatting a spreadsheet or copying data between two apps. It is a widely used test for computer use agents. Xie and colleagues released it in April 2024 with 369 tasks that run on Ubuntu, in an environment that also supports Windows and macOS, and each task is scored by checking what the machine looks like afterward, not by reading the agent’s reply.
What is in a task
The paper gives every task three parts: a natural-language instruction, a setup config that puts the virtual machine in a starting state, and a custom script that judges the result. The benchmark has 302 initial states and 134 evaluation functions. Getter functions pull out the result, such as a file or a setting, and evaluators judge it.
The tasks come from guides, videos, WikiHow, Q&A sites and courses. The paper reports these splits:
| Slice | Tasks | Share |
|---|---|---|
| Single-app | 268 | 72.6% |
| Multi-app workflow | 101 | 27.4% |
| Taken from other benchmarks (NL2Bash, Mind2Web, SheetCopilot, PPTC, GAIA) | 84 | 22.8% |
| Infeasible (deprecated or invented features) | 30 | 8.1% |
The infeasible tasks check that an agent says a task cannot be done instead of faking it. Nine student authors spent about 1,800 person-hours building the set, and two authors cross-checked each task.
How an agent interacts
The paper tests several observation settings: a 1920x1080 screenshot, a filtered accessibility tree, set-of-marks (numbered boxes drawn on the screenshot), and combinations. Actions are pyautogui code plus WAIT, FAIL and DONE, and the baselines got up to 15 steps. Everything runs in virtual machines, so an agent cannot damage the host, snapshots reset a run quickly, and many VMs can run in parallel.
What the first results showed
In the paper, humans completed over 72.36% of tasks and the best model 12.24%. Those are April 2024 numbers and are not a current ranking. Three findings still shape how people read the benchmark:
- Grounding was the main problem. The authors attribute the gap mainly to GUI grounding and operational knowledge. Over 75% of GPT-4V failures involved inaccurate mouse clicks.
- Multi-app work was hardest. The best baseline scored 6.57% on workflow tasks.
- Inputs matter. Higher screenshot resolution generally improved results.
Versions
| Version | What changed |
|---|---|
| Original (April 2024) | 369 tasks, scored by final-state scripts. |
| OSWorld-Verified (July 28, 2025) | Per the project site, fixes for community-reported examples, AWS support that brings evaluation to within an hour, and official results run by the maintainers under unified settings. |
| OSWorld 2.0 (announced June 26, 2026) | Per the 2.0 project site, 108 long-horizon tasks over seven professional domains and 31 self-hosted websites alongside desktop apps, with an average of over 250 steps and partial credit across an average of 27.25 checkpoints per task. |
The 2.0 page adds results at 150, 300 and 500 step budgets and a separate eight-check safety audit of each run. The XLANG Lab at the University of Hong Kong builds it, with Snorkel AI as a research and data partner.
How to read an OSWorld score
- Name the version. A Verified score and a 2.0 score are different tests, and Verified fixed examples in the original set.
- Check who ran it. The project site says its official results are run by the maintainers under unified settings. To appear there, a team has to have the maintainers run its agent.
- Note the step budget and observation type. A screenshot-only agent and one that reads the accessibility tree are not on equal terms, and 2.0 reports several step budgets.
- Look at the split. Multi-app and long tasks fail far more often than single-app ones, so an overall figure hides where an agent breaks.
For where OSWorld sits among other tests, see agent benchmarks. For terminal tasks, compare Terminal-Bench, and for general assistant tasks, GAIA.
Running it on your own agent
The repository supports VMware, VirtualBox, Docker, AWS, Modal and Daytona, and pip install desktop-env installs the environment without the tasks. You implement the agent interface and run it through scripts/python/run_multienv.py, which runs environments in parallel; the single-threaded run.py is marked deprecated. The project site notes that eight Google Drive tasks may need manual setup, which leaves 361.
OSWorld is a public benchmark, so it cannot tell you how an agent does on your own screens and workflows. The same idea carries over: set up a known starting state, let the agent act, and check the end state. See testing AI agents for how to build that. HoneyHive’s experiments run a dataset through a function and score each output, and tracing records the steps each run took.
FAQ
What does OSWorld measure?
Whether a multimodal agent can finish open-ended tasks on a real computer, such as editing a spreadsheet or moving files between apps. The paper has 369 tasks and scores each by checking the resulting state.
What is OSWorld-Verified?
A July 2025 update. The project site says it fixed community-reported examples and added AWS support that brings evaluation down to within an hour.
How is OSWorld 2.0 different from the original?
It has 108 long-horizon tasks that average over 250 agent steps, against under 30 in the first version, and adds partial credit. See the 2.0 project site.
How do I run OSWorld on my own agent?
Implement the agent interface and run it through the multi-environment script in the repository, on VMware, Docker, AWS or another supported backend. A place on the OSWorld-Verified leaderboard requires the maintainers to run the agent.
Why do agents fail on OSWorld?
The paper attributes the low early scores mainly to GUI grounding (finding the right thing to click) and operational knowledge of how apps work.