# Online vs offline evaluation for LLM apps

> Offline evaluation scores a fixed dataset before release. Online evaluation scores live traffic after it. They catch different failures, so most teams run both.

Source: https://www.honeyhive.ai/learn/online-vs-offline-evaluation (updated 2026-11-11)

Offline evaluation runs your application on a fixed dataset before you ship a change. Online evaluation scores real traffic after you ship. Offline answers "did this change break anything we know about?", and online answers "what is happening to users right now?". Anthropic [describes](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) automated evals as especially useful pre-launch and in CI/CD, and production monitoring as the layer that "kicks in post-launch to detect distribution drift and unanticipated real-world failures."

Both are part of [LLM evaluation](/learn/llm-evaluation). The split is about when and on what data you score, not which metric you use.

## Side by side

| | Offline | Online |
|---|---|---|
| Runs on | A fixed dataset of test cases | Live production traces, often sampled |
| When | During development and in CI, on each change | Continuously after release |
| Question it answers | Did this prompt, model or pipeline change regress? | Is quality drifting, and what are real users hitting? |
| Ground truth | Can use expected outputs | Usually none, so checks are reference-free |
| Can compare versions on identical input | Yes | No, traffic differs over time |
| Cost driver | Dataset size times judges, per run | Traffic volume times judges, ongoing |
| Main blind spot | Inputs the dataset never covered | Slower, and cannot stop a bad response in flight |

## What offline evaluation is for

A dataset holds test cases with inputs and, often, expected outputs. HoneyHive's [experiments docs](https://docs.honeyhive.ai/v2/evaluation/introduction.md) describe the loop: run your function on each datapoint, score each output, aggregate the results, then compare runs such as prompt v1 against v2 on the same input. Because the inputs are fixed, a drop in a metric points at your change.

That makes offline evals the right gate for a deploy. The same docs suggest running experiments in CI/CD to catch issues before deployment, and OpenAI [recommends](https://developers.openai.com/api/docs/guides/evaluation-best-practices) continuous evaluation that runs evals on every change. Hamel Husain [wrote](https://hamel.dev/blog/posts/evals/) that cheap assertion-style tests should run fast as you develop, with costlier model and human review and A/B tests running less often.

The weakness is coverage. Anthropic [warns](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) that automated evals can create false confidence if they do not match real usage patterns, and OpenAI [lists](https://developers.openai.com/api/docs/guides/evaluation-best-practices) datasets that do not faithfully reproduce production traffic patterns as a common design mistake. A [golden dataset](/learn/golden-dataset) only protects against failures someone already thought of.

## What online evaluation is for

Online evaluation scores traces from production. Anthropic [says](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) production monitoring reveals real user behavior at scale and catches issues that synthetic evals miss. This is where drift shows up: a provider changes a model, users start asking new things, a tool returns something unexpected. It is the scoring half of [LLM monitoring](/learn/llm-monitoring) and builds on [observability](/learn/llm-observability), since you can only score what you trace. Hamel Husain [notes](https://hamel.dev/blog/posts/evals/) that logging traces is a prerequisite for model-based and human evaluation.

Production traffic rarely comes with an expected answer, so online scoring leans on code checks (format, length, required fields) and [LLM-as-a-judge](/learn/llm-as-a-judge) graders that need only the input and output. You can still add human review on a sample.

Practical limits, taken from HoneyHive's [online evaluation docs](https://docs.honeyhive.ai/v2/monitoring/onlineevals.md) as one concrete implementation:

- Evaluators run asynchronously after ingestion, so they add no latency, but scores appear after a short delay.
- Evaluators match traces by event type, name and properties, and a sampling percentage sets how many matches get scored. The docs suggest 100% under 1K events a day, 25 to 50% for 1K to 10K, and 5 to 25% above that.
- LLM evaluators cost model calls for every evaluation, so Python evaluators can often run at 100% while LLM judges are sampled.
- Because they run after ingestion, online evaluators cannot block an unsafe response. The docs send checks that must happen during execution, such as guardrails, to client-side evaluators (see [AI guardrails](/learn/ai-guardrails)).

## Closing the loop

The two modes feed each other. Anthropic [recommends](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) converting user-reported failures into test cases so the suite reflects actual usage, and says capability evals can graduate into regression suites that run continuously. OpenAI [advises](https://developers.openai.com/api/docs/guides/evaluation-best-practices) logging everything so you can mine your logs for good eval cases and using a mix of production data in test sets. Husain [describes](https://hamel.dev/blog/posts/evals/) updating tests continuously from new failures seen in the data.

In practice the cycle is: an online score drops or a user complains, you read the failing traces ([error analysis](/learn/error-analysis-llm) is the method), you add those cases to the offline dataset, you fix the prompt or pipeline, and the offline run now guards against the same failure.

## Choosing where to start

- Before launch you have no traffic, so start offline. Build a small dataset, write graders, and run it on every change.
- After launch, add online scoring on a sample of traces for the failure modes you care about most, and alert on drops.
- If you only have capacity for one more thing, add the loop: route flagged production traces into the dataset.

## Running both in HoneyHive

In HoneyHive, [experiments](https://docs.honeyhive.ai/v2/evaluation/introduction.md) cover the offline side and [online evaluations](https://docs.honeyhive.ai/v2/monitoring/onlineevals.md) cover the online side. The online docs say evaluators run on all ingested traces that match their filters, including both production and experiment traces, so one evaluator can score both.

## FAQ

### What is the difference between offline and online evaluation?

Offline evaluation runs your app on a fixed dataset before release. Online evaluation scores real production traffic. Anthropic [describes](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) automated evals as especially useful pre-launch and in CI/CD, and production monitoring as the layer that detects drift and unanticipated real-world failures after launch.

### Do I need both?

Usually yes. Anthropic [says](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) the most effective teams pair automated evals for fast iteration with production monitoring for ground truth.

### Can online evaluation block a bad response?

Not in HoneyHive. Its [docs](https://docs.honeyhive.ai/v2/monitoring/onlineevals.md) say online evaluators run asynchronously after ingestion and point checks that must happen during execution, such as guardrails and blocking unsafe responses, to client-side evaluators.

### How do production failures become offline tests?

Anthropic [recommends](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) converting user-reported failures into test cases so the suite reflects actual usage. OpenAI [advises](https://developers.openai.com/api/docs/guides/evaluation-best-practices) logging everything so you can mine your logs for good eval cases.