Agent evaluation metrics

Agent evaluation metrics fall into five groups: outcome, tool calls, trajectory, reliability and efficiency. Each answers a different question, and each is computed in a different way, from a state check to an LLM rubric. This page lists the metrics, where each definition comes from and what it can and cannot tell you. For the process of building an eval, see how to evaluate AI agents.

Which metric answers which question

Question	Metric group	How it is computed
Did the task end in the right state?	Outcome	State check or tests, deterministic
Did the agent call the right tools correctly?	Tool calls	Compare with expected calls, deterministic
Were the steps acceptable?	Trajectory	Match against an expected path, or an LLM rubric
Does it work every time?	Reliability	Repeat trials, pass^k
What does it cost to run?	Efficiency	Count from the trace

Anthropic’s guide to agent evals defines the two things most of these metrics read from. The outcome is “the final state in the environment at the end of the trial,” and the transcript is the complete record of a trial, including outputs, tool calls, reasoning and intermediate results, also called a trace or trajectory. Outcome metrics read the environment. The other four groups read the transcript, or many transcripts.

Outcome metrics

Outcome metrics ask whether the work got done, judged by the world and not by what the agent says. The tau-bench paper compares the database state at the end of a conversation with an annotated goal state. Anthropic lists code-based graders that fit here: string match checks, binary tests such as fail-to-pass, static analysis and outcome verification.

Anthropic advises grading what the agent produced, not the path it took. For an answer with no checkable end state, such as a written reply, Google’s ADK offers response_match_score (ROUGE-1 similarity to a reference), final_response_match_v2 (an LLM-judged semantic match) and rubric_based_final_response_quality_v1 (an LLM judge applying your rubrics). See ROUGE for what the first one measures.

Tool call metrics

Tool call metrics check the individual calls. The Berkeley Function Calling Leaderboard documents a concrete method:

Check	What it does
AST match	Parses the call, verifies the function name is consistent with the expected answer, checks that all required parameters are present, and is strict on typing
Parameter check	Ensures only parameters in the function documentation are used, which flags hallucinated parameters
Parallel and multiple calls	Scored all or nothing
Executable check	Runs the generated call and checks the response; real-time matching allows a numeric result within 20% of expected
Relevance detection	When no provided function fits, the expected output is no function call

Anthropic names “tool calls verification (tools used, parameters)” as a code-based grader, so the same idea carries over to your own tools. The last row of the table matters for agents with many tools: a call that should not have happened is also a failure.

Trajectory metrics

Trajectory metrics score the steps between the request and the result. Google’s ADK defines tool_trajectory_avg_score as an exact match of the tool call trajectory, with three match types:

Match type	Passes when
EXACT (default)	Actual and expected calls match, with no extra or missing calls
IN_ORDER	Expected calls appear in the same order, and other calls may appear between them
ANY_ORDER	Expected calls appear in any order, and other calls may appear between them

Each invocation scores 1.0 on a match and 0.0 otherwise, and the criterion averages those scores across the eval case. When there is no single right path, ADK also lists LLM-judged rubrics: rubric_based_tool_use_quality_v1 and rubric_based_multi_turn_trajectory_quality_v1, both of which need rubrics you write.

Exact-path matching is the strictest option, and Anthropic’s advice to grade the product and not the path points the other way. A reasonable split is to use IN_ORDER or ANY_ORDER when only the required steps matter, and keep EXACT for flows where extra calls have a real cost. LLM-judged rubrics need calibration against people, as covered in inter-annotator agreement.

Reliability metrics

Agents vary between runs, so one success proves little (see AI agent reliability). Anthropic defines pass@k as the likelihood of at least one correct solution in k attempts and pass^k as the probability that all k trials succeed. It gives an example: a 75% per-trial rate gives roughly 42% over three trials.

The tau-bench paper introduced pass^k in June 2024 and reported that function-calling agents such as gpt-4o succeeded on under 50% of tasks, with pass^8 below 25% in retail. Use pass@k when one good answer is enough, for example when a person picks the best of several. Use pass^k when every run must work. More on the benchmark is in tau-bench.

Efficiency metrics

Efficiency metrics come straight from the trace. ADK reports four: tool_call_count_v1, inference_call_count_v1, token_usage_v1 and invocation_duration_v1. It describes them as informational, never pass or fail. Anthropic’s examples track n_turns, n_toolcalls, n_total_tokens, time_to_first_token, output_tokens_per_sec and time_to_last_token, and say latency, token usage, cost per task and error rates can be tracked on a static bank of tasks.

Treat these as constraints next to quality, not as quality. A rising tool call count on the same task bank is a signal worth reading in the trace. See time to first token and LLM cost optimization for the latency and cost side.

Quality and safety metrics

Some metrics apply to the final response of any agent. ADK lists hallucinations_v1 (LLM-judged groundedness of the response against context), safety_v1 and multi_turn_task_success_v1, which evaluates whether the agent achieves the goals of the conversation. For single-call metrics such as faithfulness, see LLM evaluation metrics.

Choosing a starting set

If your agent	Start with
Changes state somewhere (database, files, tickets)	A state check as the outcome metric, plus pass^k
Calls many tools or APIs	Tool call checks on name, parameters and relevance, plus an outcome metric
Writes free-text answers	An LLM rubric or semantic match, calibrated against people
Must follow a required sequence	IN_ORDER trajectory matching
Is expensive or slow to run	Efficiency counts tracked next to the outcome

Metrics only help if you read the runs behind them. OpenAI’s trace grading guide describes a trace as the end-to-end log of decisions, tool calls and reasoning steps and calls grading it a valuable tool for error identification at scale. For where traces come from, see agent tracing and agent observability. For test layers, see how to test AI agents.

Looking at agent metrics in HoneyHive

HoneyHive’s tracing docs describe capturing LLM calls, tool invocations and agent steps in a hierarchical view with OpenTelemetry. The Trajectory view plots each span as a bubble on a grid of event type and name against chronological step. Bubble size defaults to duration, and you can switch each group to cost or to any evaluator score attached to the spans. Clicking a bubble opens the span in the Tree view, which helps with spotting repeated tool calls and rising latency.

FAQ

What are the main metrics for evaluating an AI agent?

Five groups cover most setups: outcome (did the task end in the right state), tool calls (right tool, valid arguments), trajectory (acceptable steps), reliability (pass^k across repeated trials) and efficiency (turns, tokens, latency, cost). Anthropic's guide separates outcome from transcript and tracks both.

How do you measure tool call accuracy?

Compare each call with an expected one. The Berkeley Function Calling Leaderboard parses each call and checks the function name, required parameters, undocumented parameters and types, and it can also execute the call and check the response.

What is trajectory evaluation?

It scores the steps an agent took, not only the result. Google's ADK offers an exact match of the tool call trajectory with EXACT, IN_ORDER and ANY_ORDER match types, plus LLM-judged rubrics for tool use quality.

Should you grade the path or the outcome?

Grade the outcome first. Anthropic advises grading what the agent produced, not the path it took. Use trajectory metrics when the path itself matters, such as call order or step count.

What is pass^k and why use it?

It is the probability that all k trials of a task succeed. The tau-bench paper introduced it in June 2024 to measure reliability over multiple trials, and reported that gpt-4o function-calling agents succeeded on under 50% of tasks with pass^8 below 25% in retail.