# Agent evaluation metrics

> Agent metrics fall into outcome, tool call, trajectory, reliability and efficiency groups. Each group answers a different question and is computed differently.

Source: https://www.honeyhive.ai/learn/agent-evaluation-metrics (updated 2026-11-07)

Agent evaluation metrics fall into five groups: outcome, tool calls, trajectory, reliability and efficiency. Each answers a different question, and each is computed in a different way, from a state check to an LLM rubric. This page lists the metrics, where each definition comes from and what it can and cannot tell you. For the process of building an eval, see [how to evaluate AI agents](/learn/ai-agent-evaluation).

## Which metric answers which question

| Question | Metric group | How it is computed |
|---|---|---|
| Did the task end in the right state? | Outcome | State check or tests, deterministic |
| Did the agent call the right tools correctly? | Tool calls | Compare with expected calls, deterministic |
| Were the steps acceptable? | Trajectory | Match against an expected path, or an LLM rubric |
| Does it work every time? | Reliability | Repeat trials, pass^k |
| What does it cost to run? | Efficiency | Count from the trace |

Anthropic's [guide to agent evals](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) defines the two things most of these metrics read from. The outcome is "the final state in the environment at the end of the trial," and the transcript is the complete record of a trial, including outputs, tool calls, reasoning and intermediate results, also called a trace or trajectory. Outcome metrics read the environment. The other four groups read the transcript, or many transcripts.

## Outcome metrics

Outcome metrics ask whether the work got done, judged by the world and not by what the agent says. The [tau-bench paper](https://arxiv.org/abs/2406.12045) compares the database state at the end of a conversation with an annotated goal state. Anthropic [lists](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) code-based graders that fit here: string match checks, binary tests such as fail-to-pass, static analysis and outcome verification.

Anthropic [advises](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) grading what the agent produced, not the path it took. For an answer with no checkable end state, such as a written reply, Google's ADK [offers](https://adk.dev/evaluate/criteria/) `response_match_score` (ROUGE-1 similarity to a reference), `final_response_match_v2` (an LLM-judged semantic match) and `rubric_based_final_response_quality_v1` (an LLM judge applying your rubrics). See [ROUGE](/learn/rouge-score) for what the first one measures.

## Tool call metrics

Tool call metrics check the individual calls. The [Berkeley Function Calling Leaderboard](https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html) documents a concrete method:

| Check | What it does |
|---|---|
| AST match | Parses the call, verifies the function name is consistent with the expected answer, checks that all required parameters are present, and is strict on typing |
| Parameter check | Ensures only parameters in the function documentation are used, which flags hallucinated parameters |
| Parallel and multiple calls | Scored all or nothing |
| Executable check | Runs the generated call and checks the response; real-time matching allows a numeric result within 20% of expected |
| Relevance detection | When no provided function fits, the expected output is no function call |

Anthropic [names](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) "tool calls verification (tools used, parameters)" as a code-based grader, so the same idea carries over to your own tools. The last row of the table matters for agents with many tools: a call that should not have happened is also a failure.

## Trajectory metrics

Trajectory metrics score the steps between the request and the result. Google's ADK [defines](https://adk.dev/evaluate/criteria/) `tool_trajectory_avg_score` as an exact match of the tool call trajectory, with three match types:

| Match type | Passes when |
|---|---|
| EXACT (default) | Actual and expected calls match, with no extra or missing calls |
| IN_ORDER | Expected calls appear in the same order, and other calls may appear between them |
| ANY_ORDER | Expected calls appear in any order, and other calls may appear between them |

Each invocation scores 1.0 on a match and 0.0 otherwise, and the criterion [averages](https://adk.dev/evaluate/criteria/) those scores across the eval case. When there is no single right path, ADK also [lists](https://adk.dev/evaluate/criteria/) LLM-judged rubrics: `rubric_based_tool_use_quality_v1` and `rubric_based_multi_turn_trajectory_quality_v1`, both of which need rubrics you write.

Exact-path matching is the strictest option, and Anthropic's [advice](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) to grade the product and not the path points the other way. A reasonable split is to use IN_ORDER or ANY_ORDER when only the required steps matter, and keep EXACT for flows where extra calls have a real cost. LLM-judged rubrics need calibration against people, as covered in [inter-annotator agreement](/learn/inter-annotator-agreement).

## Reliability metrics

Agents vary between runs, so one success proves little (see [AI agent reliability](/learn/ai-agent-reliability)). Anthropic [defines](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) pass@k as the likelihood of at least one correct solution in k attempts and pass^k as the probability that all k trials succeed. It gives an example: a 75% per-trial rate gives roughly 42% over three trials.

The [tau-bench paper](https://arxiv.org/abs/2406.12045) introduced pass^k in June 2024 and reported that function-calling agents such as gpt-4o succeeded on under 50% of tasks, with pass^8 below 25% in retail. Use pass@k when one good answer is enough, for example when a person picks the best of several. Use pass^k when every run must work. More on the benchmark is in [tau-bench](/learn/tau-bench).

## Efficiency metrics

Efficiency metrics come straight from the trace. ADK [reports](https://adk.dev/evaluate/criteria/) four: `tool_call_count_v1`, `inference_call_count_v1`, `token_usage_v1` and `invocation_duration_v1`. It describes them as informational, never pass or fail. Anthropic's examples [track](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) `n_turns`, `n_toolcalls`, `n_total_tokens`, `time_to_first_token`, `output_tokens_per_sec` and `time_to_last_token`, and say latency, token usage, cost per task and error rates can be tracked on a static bank of tasks.

Treat these as constraints next to quality, not as quality. A rising tool call count on the same task bank is a signal worth reading in the trace. See [time to first token](/learn/time-to-first-token) and [LLM cost optimization](/learn/llm-cost-optimization) for the latency and cost side.

## Quality and safety metrics

Some metrics apply to the final response of any agent. ADK [lists](https://adk.dev/evaluate/criteria/) `hallucinations_v1` (LLM-judged [groundedness](/learn/llm-hallucinations) of the response against context), `safety_v1` and `multi_turn_task_success_v1`, which evaluates whether the agent achieves the goals of the conversation. For single-call metrics such as faithfulness, see [LLM evaluation metrics](/learn/llm-evaluation-metrics).

## Choosing a starting set

| If your agent | Start with |
|---|---|
| Changes state somewhere (database, files, tickets) | A state check as the outcome metric, plus pass^k |
| Calls many tools or APIs | Tool call checks on name, parameters and relevance, plus an outcome metric |
| Writes free-text answers | An LLM rubric or semantic match, calibrated against people |
| Must follow a required sequence | IN_ORDER trajectory matching |
| Is expensive or slow to run | Efficiency counts tracked next to the outcome |

Metrics only help if you read the runs behind them. OpenAI's [trace grading guide](https://developers.openai.com/api/docs/guides/trace-grading) describes a trace as the end-to-end log of decisions, tool calls and reasoning steps and calls grading it a valuable tool for error identification at scale. For where traces come from, see [agent tracing](/learn/agent-tracing) and [agent observability](/learn/agent-observability). For test layers, see [how to test AI agents](/learn/testing-ai-agents).

## Looking at agent metrics in HoneyHive

HoneyHive's [tracing docs](https://docs.honeyhive.ai/v2/tracing/introduction.md) describe capturing LLM calls, tool invocations and agent steps in a hierarchical view with OpenTelemetry. The [Trajectory view](https://docs.honeyhive.ai/v2/tracing/trajectory-view.md) plots each span as a bubble on a grid of event type and name against chronological step. Bubble size defaults to duration, and you can switch each group to cost or to any evaluator score attached to the spans. Clicking a bubble opens the span in the Tree view, which helps with spotting repeated tool calls and rising latency.

## FAQ

### What are the main metrics for evaluating an AI agent?

Five groups cover most setups: outcome (did the task end in the right state), tool calls (right tool, valid arguments), trajectory (acceptable steps), reliability (pass^k across repeated trials) and efficiency (turns, tokens, latency, cost). Anthropic's [guide](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) separates outcome from transcript and tracks both.

### How do you measure tool call accuracy?

Compare each call with an expected one. The Berkeley Function Calling Leaderboard [parses each call](https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html) and checks the function name, required parameters, undocumented parameters and types, and it can also execute the call and check the response.

### What is trajectory evaluation?

It scores the steps an agent took, not only the result. Google's ADK [offers](https://adk.dev/evaluate/criteria/) an exact match of the tool call trajectory with EXACT, IN_ORDER and ANY_ORDER match types, plus LLM-judged rubrics for tool use quality.

### Should you grade the path or the outcome?

Grade the outcome first. Anthropic [advises](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) grading what the agent produced, not the path it took. Use trajectory metrics when the path itself matters, such as call order or step count.

### What is pass^k and why use it?

It is the probability that all k trials of a task succeed. The [tau-bench paper](https://arxiv.org/abs/2406.12045) introduced it in June 2024 to measure reliability over multiple trials, and reported that gpt-4o function-calling agents succeeded on under 50% of tasks with pass^8 below 25% in retail.