What is AI agent reliability?

AI agent reliability is how consistently an agent does a task: it succeeds on every run, survives rephrased inputs and failing tools, knows when it is likely wrong, and keeps its failures small. It is a different number from accuracy. An agent can pass 90% of tasks on average and still be unusable if you cannot predict which 10% it will fail.

Princeton researchers Rabanser, Kapoor, Narayanan and colleagues proposed a framework for it in a paper first posted in February 2026. They borrow the idea from safety-critical engineering such as aviation and nuclear power, and they report that “reliability gains lag behind accuracy improvements” across the 15 models they tested. The paper finds that “outcome consistency remains low across all models,” meaning agents that can solve a task often fail to do so on repeated attempts.

The four dimensions

The paper defines twelve metrics under four dimensions. Each scores from 0 to 1, higher being better.

Dimension	Question it answers	Metrics
Consistency	Does the agent behave the same way across runs?	Outcome (succeeds or fails the same way over K runs), trajectory (same actions, same order), resource use (cost, time, calls)
Robustness	Does it hold up when conditions change?	Fault (injected tool timeouts and error responses), environment (reordered fields, altered tool interfaces), prompt (paraphrased instructions)
Predictability	Does its confidence match its accuracy?	Calibration, discrimination (do successful runs get higher confidence than failed ones), Brier score
Safety	When it fails, how bad is it?	Compliance with predefined constraints, harm severity

The authors report safety separately instead of averaging it in, because an average can hide rare severe failures. The other three dimensions make up the overall score.

Two of their results are useful when you plan tests:

• Models choose similar action types from run to run but vary the order, which the authors call “what but not when.”
• Models handle technical faults well, but how much paraphrased instructions hurt them varies substantially from model to model. The authors call this counterintuitive.

So a test suite that injects tool errors and never rewords the prompt checks the dimension that is already strongest.

Why one run is not enough

Agent output varies between runs. Anthropic’s engineers wrote that agents “are non-deterministic between runs, even with identical prompts,” that “errors compound,” and that one failing step “can cause agents to explore entirely different trajectories.” A single passing run tells you the agent can succeed, not that it will.

That is the gap between two metrics. Anthropic defines pass@k as the likelihood of at least one correct solution in k attempts, and pass^k as the probability that all k trials succeed. It suggests pass@k for tools where one success is enough and pass^k for agents where consistency is essential. The tau-bench authors introduced pass^k for exactly this reason, and reported in June 2024 that strong function-calling agents, gpt-4o among them, were “quite inconsistent (pass^8 <25% in retail).” More on the metric is in what is tau-bench.

How to measure it

The Princeton protocol is a workable template for your own agent. It used:

Test	Setup in the paper
Consistency	Five runs per task, at temperature zero where configurable
Prompt robustness	Five LLM-generated paraphrases of each task
Fault robustness	Tool faults injected with probability 0.2
Environment robustness	Medium-intensity changes to the interface, such as reordered fields

Run these on your own tasks and compare each condition with a baseline run of the same tasks. For the grading itself, Anthropic advises grading what the agent produced, not the path it took, since agents often find valid routes the eval designer did not expect. Its other warning applies to reliability work: Anthropic notes that a 0% pass rate across many trials is “most often a signal of a broken task, not an incapable agent,” so check the task and graders and read the transcripts.

Temperature zero does not fix this. The Princeton runs used it where possible and still found low outcome consistency. For related setup, see testing AI agents, agent evaluation metrics and LLM temperature.

What improves reliability

The research describes where agents are unreliable more than how to fix it, so these are the practices that engineering teams and the authors point to.

Practice	Source
Add stopping conditions such as a maximum number of iterations	Anthropic’s agent guide calls them common and says they maintain control
Test in sandboxes and add guardrails	Same guide: “extensive testing in sandboxed environments, along with the appropriate guardrails” (see AI agent guardrails)
Pause for a person at checkpoints	The guide says agents can pause for human feedback at checkpoints or blockers (see human-in-the-loop AI)
Resume instead of restarting	Anthropic’s research team built systems that resume from where the agent was when an error occurred
Set a reliability bar before launch	The Princeton authors suggest minimum consistency and safety thresholds before moving from a sandboxed pilot to production
Match the bar to the autonomy	The Princeton authors argue that a reviewed copilot can tolerate more inconsistency than a fully automated agent; one that succeeds on 90% of tasks but fails unpredictably on the rest “may be a useful assistant yet an unacceptable autonomous system”

The Princeton authors also say reliability evaluation complements human oversight, sandboxed testing and monitoring and does not replace them. They add that reliability can degrade silently as conditions drift, so re-run the evaluation periodically.

Tracking reliability in HoneyHive

Offline, experiments score runs against a dataset, and the CI regression guide fails a pull request when a metric drops past a threshold you set against a baseline run. In production, online evaluations score ingested traces asynchronously at a sampling rate you choose, and drift alerts compare a metric with earlier periods. For monitoring in general, see AI agent monitoring.

FAQ

What is AI agent reliability?

How consistently an agent does the job: the same task succeeds on every run, small changes to the input or tools do not break it, the agent knows when it is likely wrong, and failures stay small. Princeton researchers split it into consistency, robustness, predictability and safety.

How is reliability different from accuracy?

Accuracy is the average success rate. Reliability asks whether the success is repeatable. The Princeton paper reports that reliability gains lag behind accuracy gains across the models it tested.

What is pass^k?

The probability that all k trials of a task succeed. The tau-bench authors introduced it to measure reliability over multiple trials, and found in June 2024 that gpt-4o agents scored under 25% at pass^8 in the retail domain. See tau-bench.

How many times should you run each task to measure reliability?

More than once. The Princeton protocol ran each task five times, alongside five paraphrases of each instruction and injected tool faults. Anthropic recommends multiple trials because a task that passed on one run might fail on the next.

Does setting temperature to zero make an agent reliable?

No. The Princeton study ran at temperature zero where the model allowed it and still found low outcome consistency across all models. See LLM temperature.