# What is AI agent reliability?

> AI agent reliability is how consistently an agent succeeds across repeated runs, rephrased inputs and tool faults. The four dimensions, pass^k and how to measure it.

Source: https://www.honeyhive.ai/learn/ai-agent-reliability (updated 2026-11-10)

AI agent reliability is how consistently an agent does a task: it succeeds on every run, survives rephrased inputs and failing tools, knows when it is likely wrong, and keeps its failures small. It is a different number from accuracy. An agent can pass 90% of tasks on average and still be unusable if you cannot predict which 10% it will fail.

Princeton researchers Rabanser, Kapoor, Narayanan and colleagues [proposed](https://arxiv.org/html/2602.16666) a framework for it in a paper first posted in February 2026. They borrow the idea from safety-critical engineering such as aviation and nuclear power, and they [report](https://arxiv.org/html/2602.16666) that "reliability gains lag behind accuracy improvements" across the 15 models they tested. The paper [finds](https://arxiv.org/html/2602.16666) that "outcome consistency remains low across all models," meaning agents that can solve a task often fail to do so on repeated attempts.

## The four dimensions

The paper [defines](https://arxiv.org/html/2602.16666) twelve metrics under four dimensions. Each scores from 0 to 1, higher being better.

| Dimension | Question it answers | Metrics |
|---|---|---|
| Consistency | Does the agent behave the same way across runs? | Outcome (succeeds or fails the same way over K runs), trajectory (same actions, same order), resource use (cost, time, calls) |
| Robustness | Does it hold up when conditions change? | Fault (injected tool timeouts and error responses), environment (reordered fields, altered tool interfaces), prompt (paraphrased instructions) |
| Predictability | Does its confidence match its accuracy? | Calibration, discrimination (do successful runs get higher confidence than failed ones), Brier score |
| Safety | When it fails, how bad is it? | Compliance with predefined constraints, harm severity |

The authors [report](https://arxiv.org/html/2602.16666) safety separately instead of averaging it in, because an average can hide rare severe failures. The other three dimensions make up the overall score.

Two of their results are useful when you plan tests:

- Models choose similar action types from run to run but vary the order, which the authors [call](https://arxiv.org/html/2602.16666) "what but not when."
- Models handle technical faults well, but how much paraphrased instructions hurt them varies substantially from model to model. The authors [call](https://arxiv.org/html/2602.16666) this counterintuitive.

So a test suite that injects tool errors and never rewords the prompt checks the dimension that is already strongest.

## Why one run is not enough

Agent output varies between runs. Anthropic's engineers [wrote](https://www.anthropic.com/engineering/multi-agent-research-system) that agents "are non-deterministic between runs, even with identical prompts," that "errors compound," and that one failing step "can cause agents to explore entirely different trajectories." A single passing run tells you the agent can succeed, not that it will.

That is the gap between two metrics. Anthropic [defines](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) pass@k as the likelihood of at least one correct solution in k attempts, and pass^k as the probability that all k trials succeed. It [suggests](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) pass@k for tools where one success is enough and pass^k for agents where consistency is essential. The tau-bench authors [introduced](https://arxiv.org/abs/2406.12045) pass^k for exactly this reason, and reported in June 2024 that strong function-calling agents, gpt-4o among them, were "quite inconsistent (pass^8 <25% in retail)." More on the metric is in [what is tau-bench](/learn/tau-bench).

## How to measure it

The Princeton protocol is a workable template for your own agent. It [used](https://arxiv.org/html/2602.16666):

| Test | Setup in the paper |
|---|---|
| Consistency | Five runs per task, at temperature zero where configurable |
| Prompt robustness | Five LLM-generated paraphrases of each task |
| Fault robustness | Tool faults injected with probability 0.2 |
| Environment robustness | Medium-intensity changes to the interface, such as reordered fields |

Run these on your own tasks and compare each condition with a baseline run of the same tasks. For the grading itself, Anthropic [advises](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) grading what the agent produced, not the path it took, since agents often find valid routes the eval designer did not expect. Its other warning applies to reliability work: Anthropic [notes](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) that a 0% pass rate across many trials is "most often a signal of a broken task, not an incapable agent," so check the task and graders and read the transcripts.

Temperature zero does not fix this. The Princeton runs [used it](https://arxiv.org/html/2602.16666) where possible and still found low outcome consistency. For related setup, see [testing AI agents](/learn/testing-ai-agents), [agent evaluation metrics](/learn/agent-evaluation-metrics) and [LLM temperature](/learn/llm-temperature).

## What improves reliability

The research describes where agents are unreliable more than how to fix it, so these are the practices that engineering teams and the authors point to.

| Practice | Source |
|---|---|
| Add stopping conditions such as a maximum number of iterations | Anthropic's [agent guide](https://www.anthropic.com/engineering/building-effective-agents) calls them common and says they maintain control |
| Test in sandboxes and add guardrails | Same guide: "extensive testing in sandboxed environments, along with the appropriate guardrails" (see [AI agent guardrails](/learn/ai-agent-guardrails)) |
| Pause for a person at checkpoints | The guide says agents can pause for human feedback at checkpoints or blockers (see [human-in-the-loop AI](/learn/human-in-the-loop-ai)) |
| Resume instead of restarting | Anthropic's research team [built](https://www.anthropic.com/engineering/multi-agent-research-system) systems that resume from where the agent was when an error occurred |
| Set a reliability bar before launch | The Princeton authors [suggest](https://arxiv.org/html/2602.16666) minimum consistency and safety thresholds before moving from a sandboxed pilot to production |
| Match the bar to the autonomy | The Princeton authors [argue](https://arxiv.org/html/2602.16666) that a reviewed copilot can tolerate more inconsistency than a fully automated agent; one that succeeds on 90% of tasks but fails unpredictably on the rest "may be a useful assistant yet an unacceptable autonomous system" |

The Princeton authors also [say](https://arxiv.org/html/2602.16666) reliability evaluation complements human oversight, sandboxed testing and monitoring and does not replace them. They [add](https://arxiv.org/html/2602.16666) that reliability can degrade silently as conditions drift, so re-run the evaluation periodically.

## Tracking reliability in HoneyHive

Offline, [experiments](https://docs.honeyhive.ai/v2/evaluation/introduction.md) score runs against a dataset, and the [CI regression guide](https://docs.honeyhive.ai/v2/evaluation/ci-regression-detection.md) fails a pull request when a metric drops past a threshold you set against a baseline run. In production, [online evaluations](https://docs.honeyhive.ai/v2/monitoring/onlineevals.md) score ingested traces asynchronously at a sampling rate you choose, and [drift alerts](https://docs.honeyhive.ai/v2/monitoring/alerts/alerts_overview.md) compare a metric with earlier periods. For monitoring in general, see [AI agent monitoring](/learn/ai-agent-monitoring).

## FAQ

### What is AI agent reliability?

How consistently an agent does the job: the same task succeeds on every run, small changes to the input or tools do not break it, the agent knows when it is likely wrong, and failures stay small. Princeton researchers [split it](https://arxiv.org/html/2602.16666) into consistency, robustness, predictability and safety.

### How is reliability different from accuracy?

Accuracy is the average success rate. Reliability asks whether the success is repeatable. The Princeton paper [reports](https://arxiv.org/html/2602.16666) that reliability gains lag behind accuracy gains across the models it tested.

### What is pass^k?

The probability that all k trials of a task succeed. The tau-bench authors [introduced it](https://arxiv.org/abs/2406.12045) to measure reliability over multiple trials, and found in June 2024 that gpt-4o agents scored under 25% at pass^8 in the retail domain. See [tau-bench](/learn/tau-bench).

### How many times should you run each task to measure reliability?

More than once. The Princeton protocol [ran each task five times](https://arxiv.org/html/2602.16666), alongside five paraphrases of each instruction and injected tool faults. Anthropic [recommends](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) multiple trials because a task that passed on one run might fail on the next.

### Does setting temperature to zero make an agent reliable?

No. The Princeton study [ran at temperature zero](https://arxiv.org/html/2602.16666) where the model allowed it and still found low outcome consistency across all models. See [LLM temperature](/learn/llm-temperature).