# What are computer use agents?

> A computer use agent operates a screen the way a person does, by reading screenshots and sending clicks and keystrokes. How the loop works and where it is risky.

Source: https://www.honeyhive.ai/learn/computer-use-agents (updated 2026-11-05)

A computer use agent is an [AI agent](/learn/what-are-ai-agents) that operates software through its visual interface. It looks at a screenshot, picks an action such as a click, a scroll or typed text, and looks at the next screenshot to see what happened. It needs no API for the app it is driving, which is the point: it can use any program a person can.

## How the loop works

All three major providers describe the same cycle, and in all three your code does the acting:

```text
  task + tool definition
        |
        v
  model sees screenshot
        |
        v
  model proposes action
        |
        v
  your app executes it
        |
        v
  new screenshot --> back to model
        |
  no more actions: done
```

In Anthropic's version, Claude [returns](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) `tool_use` blocks and the application runs them and returns a `tool_result`, an image for screenshots and short text for other actions. Anthropic's docs say the application "runs every call in an environment you control," and that screenshots are stored in your environment rather than by Anthropic.

OpenAI's `computer` tool [returns](https://developers.openai.com/api/docs/guides/tools-computer-use) a `computer_call` with an ordered list of actions. Your app runs them, then sends a `computer_call_output` with a screenshot, and continues until the model stops returning `computer_call` items. A `completed` status only means the model finished generating the call, not that anything happened on screen.

Google's Gemini API [has the model return](https://aistudio.google.com/docs/computer-use) a `function_call` such as `click`, `type` or `scroll`. The docs say you must implement the client-side execution environment, and their examples use Playwright.

## The three APIs side by side

| | Anthropic | OpenAI | Google |
|---|---|---|---|
| Who executes actions | Your application | Your application | Your application |
| What the model returns | `tool_use` blocks, in order | `computer_call` with an `actions` array | `function_call` per action |
| What you send back | `tool_result` with a screenshot or text | `computer_call_output` with a screenshot | `function_result` with the page URL and a new screenshot |
| Coordinates | Screenshot pixels, origin top left | Map back to the original image if you downscale | Normalized 0 to 999, scaled by your client |
| Built-in confirmation signal | None; you decide | Pending safety checks that your app must handle | `safety_decision` of `require_confirmation` or `blocked` |

Sources for each cell: Anthropic's [computer use docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool), OpenAI's [computer use guide](https://developers.openai.com/api/docs/guides/tools-computer-use) and Google's [computer use docs](https://aistudio.google.com/docs/computer-use). OpenAI also offers a code execution path, where the model writes scripts that your app runs in a persistent browser or desktop.

## Computer use versus browser use

Many "browser agents" are the web-only version of this. Anthropic's [browser use tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/browser-use-tool) lets Claude work with a page through its accessibility tree, elements, forms and tabs as well as screenshots, so it can click an element reference without locating it in an image. For a whole desktop, the docs point to computer use, which works through screenshots and coordinates alone. Both run in a browser or machine your application operates; nothing runs on Anthropic's side.

## Practical limits

- **Cost grows with every step.** Anthropic [says](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) each screenshot costs roughly 1,000 to 1,800 input tokens, and long histories need pruning or caching.
- **Small targets get missed.** Anthropic [notes](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) that clicks can miss small elements on high-resolution screens and suggests a screen size such as 1280x720 as a baseline.
- **Loops need a cap.** Anthropic's example sets `max_iterations` to prevent infinite loops and "unexpected API costs." OpenAI [advises](https://developers.openai.com/api/docs/guides/tools-computer-use) step, time and cost limits.
- **A final message is not proof.** OpenAI's guidance is to check the actual outcome rather than rely on the model's answer.

## Benchmarks

[OSWorld](https://arxiv.org/abs/2404.07972) is a widely used benchmark for this. It has 369 computer tasks across Ubuntu, Windows and macOS in real applications. In the paper, submitted in April 2024, humans completed over 72.36% of tasks while the best model reached 12.24%. Those are the paper's numbers, not current ones. The maintainers [run](http://osworld-v1.xlang.ai/) official results themselves under unified settings, publish an OSWorld-Verified version (July 2025) with fixed examples, and now point to OSWorld 2.0. Check the leaderboard for current scores. Read a score with its settings: which version, how many steps allowed, and who ran it.

## Security and oversight

A screen is untrusted input, so a web page, email or document the agent opens can try to steer it. The three providers give nearly the same advice:

- **Isolate it.** Anthropic recommends a [dedicated VM or container](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) with minimal privileges, and OpenAI an isolated browser or VM with an allow list of sites and actions. Google [recommends](https://aistudio.google.com/docs/computer-use) a sandboxed VM, container or limited browser profile.
- **Keep secrets out.** Anthropic [advises](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) against giving the model access to sensitive data such as login information.
- **Restrict where it can go.** All three suggest allowlists for sites.
- **Put a person on consequential steps.** Anthropic [lists](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) financial transactions, accepting cookies and agreeing to terms of service. OpenAI [says](https://developers.openai.com/api/docs/guides/tools-computer-use) typing sensitive data into a form counts as transmission, so it needs confirmation. See [human-in-the-loop AI](/learn/human-in-the-loop-ai).
- **Log everything.** Google [recommends](https://aistudio.google.com/docs/computer-use) logging prompts, screenshots, proposed actions, safety responses and executed actions.

Anthropic also [says](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) its classifiers scan screenshots for prompt injection but that the precautions stay important regardless. For the wider threat, see [prompt injection](/learn/prompt-injection) and [AI agent guardrails](/learn/ai-agent-guardrails).

## Tracing and testing a computer use run

Each step is a model call plus an action and a screenshot, so a failed task is usually one wrong step in a long chain. Capturing the screenshot, the proposed action and the result at every step is what makes a run debuggable, and it matches Google's logging advice. [Testing AI agents](/learn/testing-ai-agents) covers scoring end state across runs, and [agent observability](/learn/agent-observability) covers recording the steps. HoneyHive's [docs](https://docs.honeyhive.ai) cover tracing agent runs.

## FAQ

### Is a computer use agent the same as a browser agent?

A browser agent is the web-only case. Anthropic's [browser use tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/browser-use-tool) works on a page's structure (accessibility tree, elements, forms, tabs) as well as screenshots, while its [computer use tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) works through screenshots and coordinates and can drive a whole desktop.

### Does the model run the clicks itself?

No. In the [OpenAI](https://developers.openai.com/api/docs/guides/tools-computer-use), [Anthropic](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) and [Google](https://aistudio.google.com/docs/computer-use) APIs the model only proposes actions. Your application executes them in a browser, VM or desktop you control and sends back a new screenshot.

### How is a computer use agent evaluated?

With benchmarks that run tasks in a real operating system and check the end state, such as [OSWorld](https://arxiv.org/abs/2404.07972). Score the outcome, not the model's final message. See [testing AI agents](/learn/testing-ai-agents).

### What is the main security risk?

Prompt injection through what is on screen. OpenAI's docs say to [treat screen content as untrusted](https://developers.openai.com/api/docs/guides/tools-computer-use), and Anthropic [warns](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) that Claude may follow commands found in content even when they conflict with your instructions.