What is tau-bench? Tool-agent-user benchmark and pass^k
tau-bench tests AI agents in simulated customer service chats with tools and policies. See how it scores, what pass^k means and how tau2 and tau3 differ.
tau-bench (written τ-bench) is a benchmark from Sierra that tests whether an AI agent can complete tasks in a conversation with a user while using tools and following company rules. The paper (Yao, Shinn, Razavi and Narasimhan, submitted June 2024) simulates a user with a language model, gives the agent domain API tools and a policy, and checks the database at the end. It also introduced pass^k, a measure of reliability across repeated trials.
Why it was built
Sierra’s launch post (June 2024) says earlier agent benchmarks test a single round of exchange and focus on average performance, not reliability. Real support conversations take several turns, and the agent has to gather information and respect rules along the way.
The paper reports that even function-calling agents such as gpt-4o succeed on fewer than 50% of tasks, and that pass^8 is below 25% in retail (arXiv abstract).
How a task is scored
Each task puts an agent and a simulated user in a domain. Per the paper:
- The domain has a database, API tools and a Markdown policy that serves as the agent’s system prompt.
- The user simulator gets the task instruction and the conversation, but cannot see the agent’s tool calls. The paper used gpt-4-0613 for it. The episode ends when the user emits “###STOP###” or after 30 agent actions.
- The reward is 1 only if two checks pass: the final database is identical to the unique ground-truth database, and the agent’s replies contain the required information.
Scoring the database state instead of the transcript makes the check automatic. The paper also notes that a reward of 1 is necessary but not sufficient for good behavior. An agent could, for example, issue a return without the explicit user confirmation the policy requires.
| Retail | Airline | |
|---|---|---|
| Tasks | 115 | 50 |
| Users | 500 | 500 |
| Catalog | 50 products, 1,000 orders | 300 flights, 2,000 reservations |
| Tools | 7 write, 8 non-write | 6 write, 7 non-write |
Source for the table: paper.
pass^k
pass^k is the probability that all k independent trials of a task succeed, averaged over tasks. With n trials per task and c successes, the paper gives it as the average of C(c,k) / C(n,k). It is the opposite of pass@k, which asks whether at least one trial succeeds.
Sierra’s example: a 25% pass^8 means a 25% chance that the agent resolves 8 cases of the same issue with different customers. The tau-bench repository reports Pass^1 through Pass^4 per agent strategy, and its example scores fall as k rises.
Versions
| Name | What changed | Source |
|---|---|---|
| tau-bench (June 2024) | Retail and airline, pass^k | taubench.com |
| tau2-bench (June 2025) | Telecom domain with dual control: agent and user both use tools in a shared system. Adds a task generator and an environment-bound user simulator | Paper |
| Task audit (February 2026) | More than 50 airline and retail tasks corrected | taubench.com |
| tau-knowledge (March 2026) | Banking domain with a knowledge base of about 700 documents | taubench.com |
| tau-voice (March 2026) | Real-time, full-duplex voice across retail, airline, telecom and banking | taubench.com |
As of November 2026, the taubench.com leaderboards are split into tau3-Banking (text, knowledge retrieval), tau3-Voice and tau2-bench (text: retail, airline, telecom). The original repository’s notice says its tasks are not updated and points to tau3-bench for fixed tasks and new domains (GitHub). Check the leaderboard for current scores; they change often.
Reading a score
- Name the version and domain. The tau2-bench README says 2.0 scores are not comparable with the 1.x leaderboard, which has its own tab.
- Name k. A Pass^1 and a Pass^4 for the same agent can differ a lot.
- Name the user simulator. The original repository defaults to gpt-4o, the paper used gpt-4-0613, and the tau2-bench README lists other defaults for voice. A different simulator changes the conversations.
- Remember the scope. The domains are customer service with a written policy. A high score does not show an agent will do well on coding or open research; see SWE-bench for software tasks.
Running it
The tau2-bench repository installs with uv sync and runs with commands such as tau2 run --domain airline --agent-llm ... --user-llm ... --num-trials 1 --num-tasks 5. Results go to taubench.com through the submission guide linked there.
Using the idea on your own agent
The pattern carries over: simulate users, check the end state, repeat each task and report pass^k. For how to build such checks, see AI agent evaluation. HoneyHive’s docs describe tracing and evaluating agents with experiments and evaluators, and its Thread View shows multi-turn conversations between users, assistants and tools.
FAQ
What does tau-bench measure?
It measures whether an agent can finish a task in a multi-turn conversation with a simulated user while calling domain API tools and following a written policy. The result is checked against the final database state.
What is pass^k?
pass^k is the chance that an agent succeeds on all k trials of the same task, averaged across tasks. It falls as k grows, so it shows reliability that a single-run score hides.
What is the difference between tau-bench, tau2-bench and tau3-bench?
tau-bench (2024) has retail and airline. tau2-bench (2025) adds a telecom domain where the user also acts on the system. The taubench.com site groups later work, such as banking and voice, under tau3 leaderboards. The tau2-bench repository says 2.0 scores are not comparable with 1.x.