How to evaluate AI agents

To evaluate an AI agent, give it a set of realistic tasks, run each task more than once, and grade two things: whether the task ended in the right state, and whether the steps it took were acceptable. Then track how consistently it succeeds, not only how often. The rest is choosing graders, building the task set and reading what the agent actually did.

This is a case of LLM evaluation where the thing under test acts over many steps. For what an agent is, see what are AI agents.

Why agents need their own evals

A single LLM call has one input and one output. An agent calls tools, reads the results and decides again. Google’s ADK docs note that because models are probabilistic, deterministic pass or fail assertions are often unsuitable, and that agents need evaluation of both the final output and the trajectory.

The thing you score is also bigger than the model. The Princeton paper on emerging trends in agent evaluation says an agent’s score depends on its scaffold and environment as well as its model, and that choices about prompts, tools, compute budget and statistics can change scores for the same model and benchmark.

The vocabulary

Anthropic’s guide to agent evals (January 2026) defines the terms most teams use:

Term	Meaning
Task	A single test with defined inputs and success criteria
Trial	One attempt at a task. Several are run because outputs vary
Grader	Logic that scores some aspect of the agent’s performance
Transcript	The complete record of a trial: outputs, tool calls, reasoning and intermediate results (also called a trace or trajectory)
Outcome	The final state of the environment at the end of the trial
Harness	The infrastructure that runs evals end to end

The distinction that matters most is outcome versus transcript. Outcome means the state of the environment, not what the agent says it did. The tau-bench paper handles this by comparing the database state at the end of a conversation with an annotated goal state.

What to measure

Layer	Question	Example checks
Outcome	Did the task end in the right state?	Task completion, error rate (IBM); end state of the environment (Anthropic)
Final response	Is the answer correct, grounded and safe?	LLM-judged match to a reference, groundedness and safety criteria (ADK)
Trajectory	Were the steps and tool calls acceptable?	Exact match of the tool call trajectory, or rubric-based tool use quality (ADK)
Tool calls	Were the arguments valid?	Wrong function name, missing required parameters, hallucinated parameter (IBM)
Efficiency	What did it cost?	Tool call count, model call count, tokens and duration, which ADK reports but never uses to pass or fail a case

IBM adds that intermediate actions, such as database queries or API calls, each need to be evaluated separately.

Choosing graders

Anthropic lists three kinds, with these trade-offs (source):

• Code-based: fast, cheap, objective, reproducible and easy to debug, but brittle to valid variations that do not match the expected pattern.
• Model-based: flexible, scalable and able to capture nuance, but non-deterministic, more expensive, and in need of calibration against human graders. See inter-annotator agreement.
• Human: the gold standard for quality, but expensive and slow.

The same guide advises grading what the agent produced rather than the exact path it took, giving partial credit on multi-step tasks, and letting an LLM judge return “Unknown” instead of forcing a verdict.

Measure reliability, not only success

Agents give different results on the same task. Anthropic separates two metrics (source):

• pass@k: the likelihood of at least one correct solution in k attempts.
• pass^k: the probability that all k trials succeed.

At a 75% per-trial success rate, the chance of passing all 3 trials is about 42%. The tau-bench authors, who introduced pass^k, reported that function-calling agents such as gpt-4o succeeded on fewer than 50% of tasks, with pass^8 below 25% in the retail domain. For an agent that talks to customers, pass^k is the number closer to what users experience.

Advice by agent type

Anthropic’s guide gives different starting points (source):

Agent	Grading approach
Coding	Deterministic graders fit naturally. Use unit tests for correctness, plus an LLM rubric for code quality if needed. See SWE-bench
Conversational	The quality of the interaction is part of what you evaluate. Often a second LLM plays the user. Check end state, transcript limits such as turn count, and LLM rubrics
Research	Quality can only be judged relative to the task. Combine groundedness, coverage and source quality checks, and calibrate LLM rubrics against experts
Computer use	Run the agent in a real or sandboxed environment

How to start

From Anthropic’s roadmap:

1. Start early with 20 to 50 simple tasks drawn from real failures. Convert user-reported failures into test cases.
2. Write unambiguous tasks. A good task is one where two domain experts would independently reach the same pass or fail verdict. A 0% pass rate across many trials is most often a signal of a broken task.
3. Test both directions: when a behavior should happen and when it should not.
4. Run each trial in a stable, clean environment so trials do not affect each other.
5. Read the transcripts. Failures should seem fair.
6. Watch for saturation. An eval at 100% tracks regressions (see how to test AI agents for the layers) but gives no signal for improvement.

Anthropic also splits suites into capability evals, which start at a low pass rate, and regression evals, which should sit near 100%. Strong capability tasks can graduate into the regression suite.

Common ways evals mislead

• Broken graders. Anthropic reports that Opus 4.5 scored 42% on CORE-Bench until a researcher fixed rigid grading, ambiguous task specs and stochastic tasks and switched to a less constrained scaffold, after which it reached 95% (source).
• Gaming and contamination. The Princeton paper says agents can exploit bugs in tasks and graders, and that tool access opens new routes to contamination. It recommends inspecting agent logs, not just outcomes (source).
• Hidden setup differences. Scaffold, tools and compute budget move scores, so record them with each result (source).

Where traces fit

Trajectory grading needs the trace of each run, and online monitoring needs traces from production. See agent observability for what a trace records.

Running agent evals in HoneyHive

A HoneyHive experiment has three parts: a function under test such as an agent, a dataset of test cases, and evaluators that score outputs. evaluate() runs the function on each datapoint with automatic tracing, scores the outputs, aggregates the metrics and shows them in the dashboard. The run_id and datapoint_id link traces so you can compare the same test case across runs. Setup is in the HoneyHive evaluation docs.

FAQ

How is evaluating an agent different from evaluating an LLM?

An agent takes many steps and uses tools, so you grade more than one answer. Google's ADK docs say to assess both the final response and the trajectory, the steps the agent took before replying. The Princeton agent evaluation paper adds that a score depends on the scaffold and environment as well as the model. See LLM evaluation for the single-call case.

What is pass^k?

The probability that all k trials of a task succeed. Anthropic contrasts it with pass@k, which counts a task as solved if any of k trials succeeds. The metric was introduced in the tau-bench paper.

How many test cases do I need to start?

Anthropic says 20 to 50 simple tasks drawn from real failures is a great start.

Should I grade the path the agent took?

Mostly grade the outcome. Anthropic advises grading what the agent produced rather than the exact path. Google's ADK offers trajectory checks such as an exact match of the tool call trajectory, for cases where the path itself matters.