Evaluations Across the Agent Development Lifecycle

The most common failure mode we see in agent evaluation isn't bad evaluators. It's applying the right evaluators at the wrong stage.

‍

Teams that over-test during prototyping burn months before they've even validated the approach. Teams that under-invest in production evaluation miss the failure modes that actually matter — the ones you can't anticipate from a dev environment.

‍

In our previous post on the ADLC, we laid out the six stages of shipping agents at scale. This post goes deeper on the dimension that runs through all of them: how evaluation rigor, cost-quality trade-offs, and evaluator selection should evolve as an agent matures.

‍

The progression

There's a default intuition borrowed from traditional software: invest heavily in testing before deployment, keep production monitoring lightweight. For agents, this is backwards.

‍

The test sets you build in development are approximations of reality — your best guess at what users will do, constructed before you've seen what users actually do. No matter how thoughtfully you curate them, they represent your imagination of the input distribution, not the distribution itself. For deterministic software, that gap is manageable. For agents operating over an effectively unbounded input space with non-deterministic behavior, it's fundamental.

‍

Production is where ground truth lives. It's where you discover failure modes you couldn't have invented, learn which edge cases occur at frequency, and get empirical signal on whether your evaluators measure the right things. That's where the serious evaluation investment belongs.

Development evaluation is cheap, fast regression prevention — protecting against things you already know matter. Pre-production is a confidence gate, not a discovery mechanism. Production is where you learn. The rest of this post is about what each stage actually requires.

‍

Prototyping: optimize for iteration speed

You're answering one question: is this approach fundamentally sound?
‍

At this stage, missing some failure modes is fine — the agent isn't anywhere near users. What's costly is false positives: evaluators incorrectly flagging acceptable outputs, forcing you to investigate non-issues when you should be iterating on the core approach.
‍

This means simple, fast evaluators with high recall:

Rule-based checks for obvious failures (format violations, empty responses, safety triggers)
Small curated test sets (10–50 examples) representing the input distribution, not just the happy path
Manual spot-checking by engineers — your judgment is the evaluator here

The temptation is to reach for LLM-based judges early because they feel more thorough. Resist it. At this stage, an LLM judge adds latency and cost to your feedback loop without meaningfully changing your decisions. You're not trying to measure quality precisely — you're trying to determine whether the entire approach is worth pursuing. A fast heuristic that gives you a directional answer in under a second is more valuable than a nuanced judge that takes five seconds and $0.01 per call.
‍

What is worth investing in early: tracing infrastructure. Capture every interaction, even throwaway experiments. The behavioral dataset you build passively during prototyping becomes the foundation for every evaluation stage that follows.

‍

Development: cheap, fast regression prevention

The role of evaluation in development is narrower than most teams assume. You're not trying to discover new failure modes here — production will do that far more effectively. You're trying to ensure that changes don't break things you've already fixed.
‍

Your test set grows to 100–500 examples and automated evals run on every commit. The reason per-commit evaluation matters for agents: non-determinism means a change that improves average performance can simultaneously introduce a regression on a specific class of inputs. Without automated checks on every commit, these regressions hide until production, where they're 10x more expensive to diagnose.

‍

Tiered evaluation keeps this sustainable. Most test cases don't need your most expensive evaluator. A cheap heuristic can filter out 60–70% of obvious failures before an LLM judge ever sees them.

The efficiency constraint matters here because this is where you're running the highest volume of evaluations — every commit, across every agent. If your eval suite takes 20 minutes, engineers stop running it, and you lose the regression detection that justifies per-commit evaluation in the first place. Optimize for speed and cost. Save the rigor for production.
‍

One thing to internalize early: every evaluator you build at this stage encodes assumptions about what "good" looks like — assumptions you're forming before you have real user signal. That's unavoidable, but recognize it. These evaluators are hypotheses. Production will tell you which ones were right.

‍

Pre-production: a confidence gate, not a discovery engine

The question here is organizational: are we confident enough to put this in front of users?

‍

Pre-production validation is a gate. You're proving minimum readiness to stakeholders — engineering, product, risk, compliance — not trying to exhaustively cover the input space. You'll never achieve that in a dev environment anyway.
‍

What changes at this stage:
‍

Test set scale — 1,000+ examples, including adversarial scenarios and edge cases deliberately constructed to break the agent
Evaluator quality — frontier models or domain-specific fine-tuned judges on critical paths. Multiple evaluators cross-validating important outputs.
Human review — domain experts on genuinely ambiguous cases and anything with regulatory implications
Red-teaming — dedicated adversarial testing for prompt injection, data exfiltration, jailbreaks, and domain-specific failure modes

The organizational dimension matters as much as the technical one. Risk, compliance, and business stakeholders need to develop confidence independently — not by watching an engineer run a demo. Evaluation dashboards, sample interaction logs, and failure-mode documentation are what turn a six-month sign-off into a two-week one.

‍

Don't mistake this stage for comprehensive coverage. You're establishing a floor of quality and safety, not a ceiling. The real evaluation happens when the agent meets actual users.

‍

Production: where the serious evaluation investment goes

This is where most teams under-invest, and it's where the highest-value evaluation happens.

‍

Every interaction with a real user generates signal you couldn't have manufactured in development. The long tail of inputs, the unexpected phrasings, the multi-turn conversations that drift into territory your test set never imagined — this is the distribution that matters, and you only see it in production.

‍

Risk-weighted sampling gives you coverage where it matters:

‍

100% evaluation for high-stakes interactions — financial transactions, PII-adjacent queries, medical or legal guidance, anything with regulatory exposure
10–20% sampling for medium-risk interactions — enough to detect systematic quality shifts within hours
1–5% sampling for low-risk, high-volume interactions — watching for distributional anomalies, not evaluating individual responses

The sampling rates should be calibrated to your error budget and detection latency requirements. If you need to detect a 5% quality degradation within 4 hours on a query category that gets 1,000 requests/day, you can work backward to the minimum sample rate that gives you statistical power. Anything less is guesswork; anything more is waste.

‍

But sampling is just the monitoring layer. The deeper investment is in what you do with production signal:

‍

Evaluator validation. You now have real user feedback — task completion rates, escalation frequency, satisfaction signals — and you can measure whether your eval scores actually correlate with these outcomes. This is the moment your development-stage evaluators go from hypotheses to empirically tested instruments. When an LLM judge scores responses highly that users consistently flag as unhelpful, you've found evaluator drift — and that's one of the most valuable discoveries you can make, because it means every eval you've been running has been measuring the wrong thing.

‍

Failure mode discovery. The monitoring signals that matter shift over time. Early in deployment: acute failures (errors, refusals, confidently wrong outputs). As the agent stabilizes: slow drift — gradual quality degradation invisible in any single interaction but clear in aggregate over days or weeks. These slow-drift patterns are almost impossible to catch in development. They emerge from the interaction between your agent and the true input distribution over time.

‍

Test set curation. Production interactions that surface new failure modes feed directly back into your development eval suite — not as synthetic test cases you imagined, but as real examples of how the agent actually fails. This is how your dev test set improves: not by engineers brainstorming edge cases, but by production continuously supplying them.

‍

Distributed tracing across the full agent execution graph is what makes all of this tractable. When a multi-step agent produces a bad output, you need to see which tool call returned unexpected data, where the reasoning chain diverged, and what the downstream effects were.

‍

Post-incident: unlimited rigor

When an agent fails in production, the evaluation question changes completely: why did this happen, and how do we guarantee it doesn't happen again?

‍

Cost is not a constraint. A single production failure can be orders of magnitude more expensive than any amount of post-incident analysis.

‍

What this looks like in practice:

‍

Trace reconstruction — walk the full execution graph of the failed interaction. Which tool call returned unexpected data? Where did the reasoning chain diverge? What was the downstream impact?
Reproduction — recreate the failure conditions and validate that you can trigger the same behavior reliably. If you can't reproduce it, you can't prove you've fixed it.
Variation testing — evaluate against mutations of the failure scenario. If the agent failed on this input, does it also fail on semantically similar inputs? How broad is the failure class?
Regression test creation — the failure scenario and its variations become permanent additions to your dev eval suite, running on every subsequent commit. Non-negotiable.

The correlation gaps you surfaced in production monitoring — where eval scores diverged from user outcomes — also get resolved here. Retune LLM judges, adjust heuristic thresholds, add evaluation dimensions that map to the failure modes you're actually seeing.

Budget allocation across stages

A rough framework for how evaluation spend should distribute in a mature agent program:

‍

Prototyping: Negligible. Manual effort, minimal tooling cost.
Development: ~15–20% of budget. High volume, low cost per eval. Optimize for speed.
Pre-production: ~10–15% of budget. Lower volume, higher cost per eval. Enough to clear the confidence gate.
Production: ~50–60% of budget. This is where you learn what matters. Sampling infrastructure, evaluator validation against user outcomes, failure mode discovery, and continuous test set curation.
Post-incident: Uncapped. Funded from incident response, not the evaluation line item.

This feels counterintuitive if you're coming from traditional software testing, where the goal is to catch everything before deployment. For agents, that goal is unrealistic — the input space is too large, the behavior is too non-deterministic, and the failure modes that matter most are the ones you couldn't have anticipated. Invest where the signal is.

‍

The evaluator lifecycle

One pattern worth making explicit: evaluators follow their own development lifecycle, parallel to the agent's.

‍

During prototyping, your evaluators are simple heuristics — and that's correct. During development, you layer in LLM judges based on hypotheses about quality. In production, you validate those hypotheses against real user outcomes. Post-incident, you iterate on evaluators that proved miscalibrated.

‍

This is the through-line of the whole framework. Development evaluation is where you deploy evaluators. Production is where you validate them. The teams that treat evaluators as fixed infrastructure — built once, trusted forever — end up optimizing for scores that have quietly decoupled from reality. The teams shipping reliably at scale treat evaluator development with the same rigor they apply to agent development: versioned, tested, and continuously improved based on production evidence.

‍

The tooling that supports this needs evaluation primitives that work across stages — from lightweight heuristics in CI to production sampling to post-incident deep dives — in a single system. This is what we've built at HoneyHive.

About the author:

Mohak Sharma

Co-Founder & CEO

Evaluations Across the Agent Development Lifecycle

About the author:

Mohak Sharma

Join our monthly newsletter