What are computer use agents?

A computer use agent is an AI agent that operates software through its visual interface. It looks at a screenshot, picks an action such as a click, a scroll or typed text, and looks at the next screenshot to see what happened. It needs no API for the app it is driving, which is the point: it can use any program a person can.

How the loop works

All three major providers describe the same cycle, and in all three your code does the acting:

  task + tool definition
        |
        v
  model sees screenshot
        |
        v
  model proposes action
        |
        v
  your app executes it
        |
        v
  new screenshot --> back to model
        |
  no more actions: done

In Anthropic’s version, Claude returns tool_use blocks and the application runs them and returns a tool_result, an image for screenshots and short text for other actions. Anthropic’s docs say the application “runs every call in an environment you control,” and that screenshots are stored in your environment rather than by Anthropic.

OpenAI’s computer tool returns a computer_call with an ordered list of actions. Your app runs them, then sends a computer_call_output with a screenshot, and continues until the model stops returning computer_call items. A completed status only means the model finished generating the call, not that anything happened on screen.

Google’s Gemini API has the model return a function_call such as click, type or scroll. The docs say you must implement the client-side execution environment, and their examples use Playwright.

The three APIs side by side

	Anthropic	OpenAI	Google
Who executes actions	Your application	Your application	Your application
What the model returns	tool_use blocks, in order	computer_call with an actions array	function_call per action
What you send back	tool_result with a screenshot or text	computer_call_output with a screenshot	function_result with the page URL and a new screenshot
Coordinates	Screenshot pixels, origin top left	Map back to the original image if you downscale	Normalized 0 to 999, scaled by your client
Built-in confirmation signal	None; you decide	Pending safety checks that your app must handle	safety_decision of require_confirmation or blocked

Sources for each cell: Anthropic’s computer use docs, OpenAI’s computer use guide and Google’s computer use docs. OpenAI also offers a code execution path, where the model writes scripts that your app runs in a persistent browser or desktop.

Computer use versus browser use

Many “browser agents” are the web-only version of this. Anthropic’s browser use tool lets Claude work with a page through its accessibility tree, elements, forms and tabs as well as screenshots, so it can click an element reference without locating it in an image. For a whole desktop, the docs point to computer use, which works through screenshots and coordinates alone. Both run in a browser or machine your application operates; nothing runs on Anthropic’s side.

Practical limits

• Cost grows with every step. Anthropic says each screenshot costs roughly 1,000 to 1,800 input tokens, and long histories need pruning or caching.
• Small targets get missed. Anthropic notes that clicks can miss small elements on high-resolution screens and suggests a screen size such as 1280x720 as a baseline.
• Loops need a cap. Anthropic’s example sets max_iterations to prevent infinite loops and “unexpected API costs.” OpenAI advises step, time and cost limits.
• A final message is not proof. OpenAI’s guidance is to check the actual outcome rather than rely on the model’s answer.

Benchmarks

OSWorld is a widely used benchmark for this. It has 369 computer tasks across Ubuntu, Windows and macOS in real applications. In the paper, submitted in April 2024, humans completed over 72.36% of tasks while the best model reached 12.24%. Those are the paper’s numbers, not current ones. The maintainers run official results themselves under unified settings, publish an OSWorld-Verified version (July 2025) with fixed examples, and now point to OSWorld 2.0. Check the leaderboard for current scores. Read a score with its settings: which version, how many steps allowed, and who ran it.

Security and oversight

A screen is untrusted input, so a web page, email or document the agent opens can try to steer it. The three providers give nearly the same advice:

• Isolate it. Anthropic recommends a dedicated VM or container with minimal privileges, and OpenAI an isolated browser or VM with an allow list of sites and actions. Google recommends a sandboxed VM, container or limited browser profile.
• Keep secrets out. Anthropic advises against giving the model access to sensitive data such as login information.
• Restrict where it can go. All three suggest allowlists for sites.
• Put a person on consequential steps. Anthropic lists financial transactions, accepting cookies and agreeing to terms of service. OpenAI says typing sensitive data into a form counts as transmission, so it needs confirmation. See human-in-the-loop AI.
• Log everything. Google recommends logging prompts, screenshots, proposed actions, safety responses and executed actions.

Anthropic also says its classifiers scan screenshots for prompt injection but that the precautions stay important regardless. For the wider threat, see prompt injection and AI agent guardrails.

Tracing and testing a computer use run

Each step is a model call plus an action and a screenshot, so a failed task is usually one wrong step in a long chain. Capturing the screenshot, the proposed action and the result at every step is what makes a run debuggable, and it matches Google’s logging advice. Testing AI agents covers scoring end state across runs, and agent observability covers recording the steps. HoneyHive’s docs cover tracing agent runs.

FAQ

Is a computer use agent the same as a browser agent?

A browser agent is the web-only case. Anthropic's browser use tool works on a page's structure (accessibility tree, elements, forms, tabs) as well as screenshots, while its computer use tool works through screenshots and coordinates and can drive a whole desktop.

Does the model run the clicks itself?

No. In the OpenAI, Anthropic and Google APIs the model only proposes actions. Your application executes them in a browser, VM or desktop you control and sends back a new screenshot.

How is a computer use agent evaluated?

With benchmarks that run tasks in a real operating system and check the end state, such as OSWorld. Score the outcome, not the model's final message. See testing AI agents.

What is the main security risk?

Prompt injection through what is on screen. OpenAI's docs say to treat screen content as untrusted, and Anthropic warns that Claude may follow commands found in content even when they conflict with your instructions.