# How to test LLM applications

> Test an LLM application in layers: unit-test the code around the model with a stub, grade real outputs on a fixed dataset, then check production traffic.

Source: https://www.honeyhive.ai/learn/llm-testing (updated 2026-11-12)

To test an LLM application, split it into parts and test each the way it behaves. Unit-test the code around the model (prompt building, parsing, validation) with a stub in place of the model. Grade real model outputs against a fixed dataset with graders that tolerate variation. Then check production traffic for what the dataset missed. Hamel Husain [describes](https://hamel.dev/blog/posts/evals/) three levels of this kind (unit tests, human and model evaluation, A/B tests) and says each costs more than the one before it, which sets how often to run it.

This page is the overall approach. For the CI gate that reruns a dataset on every change, see [LLM regression testing](/learn/llm-regression-testing). For agents, see [how to test AI agents](/learn/testing-ai-agents). For metrics and judges, see [LLM evaluation](/learn/llm-evaluation).

## What to test at each layer

| Layer | What it checks | How to grade | When to run |
|---|---|---|---|
| Code around the model | Prompt templates, parsing, validation, error handling | Ordinary assertions with the model stubbed | Every commit |
| Model behavior | Quality on typical, edge and adversarial inputs | [Code-based, model-based or human graders](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) over a fixed dataset | On every prompt, model or code change |
| Safety and abuse | Harmful, off-topic or manipulated inputs | Adversarial prompts and a content classifier or judge | Before release and on a schedule |
| Production | Real user inputs the dataset never covered | Sampled traces, user feedback, [A/B tests](https://hamel.dev/blog/posts/evals/) | Continuously |

## Test the code around the model without the model

Most of an LLM application is ordinary code, and it should have ordinary tests. Pydantic AI's testing guide [recommends](https://pydantic.dev/docs/ai/guides/testing/) replacing the model with a stand-in so tests avoid the usage, latency and variability of real LLM calls. Its `TestModel` is described as the simplest and fastest way to exercise most of your application code, and `FunctionModel` lets you write your own response logic. Setting `ALLOW_MODEL_REQUESTS` to `False` [guards](https://pydantic.dev/docs/ai/guides/testing/) against accidental real requests during tests.

The guide is explicit that these [unit tests check your own code, not answer quality](https://pydantic.dev/docs/ai/guides/testing/). Other frameworks have their own stand-ins, and the idea transfers: a test that needs a live model call is an eval, and it belongs in the next layer. If your application depends on a fixed output shape, test that the parser handles malformed output as well as valid output; see [structured output](/learn/structured-output-llm).

Hamel Husain [describes](https://hamel.dev/blog/posts/evals/) the assertions he uses at this level as checks "like you would write in pytest" that run fast and cheaply. He scopes them by feature and scenario, such as one match, several matches and no matches, and adds generic checks like a regex that catches exposed UUIDs in output.

## Grade real outputs on a fixed dataset

For the model's own behavior, run the application on a fixed set of inputs and grade each output. Anthropic's guide [compares](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) three kinds of grader.

| Grader | Strengths | Weaknesses |
|---|---|---|
| Code-based (string match, schema checks, static analysis) | Fast, cheap, objective, reproducible, easy to debug | Can reject valid answers that differ from the expected pattern; limited for some subjective tasks |
| Model-based (rubric scoring, pairwise comparison) | Flexible, scales, handles open-ended output | Results vary between runs, costs more, needs calibration against humans |
| Human (expert review, spot checks) | The gold standard, calibrates the model graders | Expensive and slow |

Use the cheapest grader that can tell a good output from a bad one. Anthropic's docs [advise automating grading](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) where possible and say more questions with slightly lower-signal automated grading beat fewer hand-graded ones. Where you use a model as the grader, the same docs say it is [generally best practice](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) to use a different model from the one that produced the output, and OpenAI [advises](https://developers.openai.com/api/docs/guides/evaluation-best-practices) scaling up an LLM judge only once it consistently agrees with human annotations. See [LLM as a judge](/learn/llm-as-a-judge).

Pick metrics that fit the task. Eugene Yan [argues](https://eugeneyan.com/writing/evals/) that accuracy is too coarse for classification and extraction and that recall and precision tell you more, and that off-the-shelf evals barely correlate with application-specific performance. OpenAI [lists](https://developers.openai.com/api/docs/guides/evaluation-best-practices) overly generic metrics such as perplexity or BLEU as a pitfall. Define success criteria first. Anthropic's docs say criteria should be [specific, measurable, achievable and relevant](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests). For metric choices see [LLM evaluation metrics](/learn/llm-evaluation-metrics).

## Build the dataset from real failures

OpenAI's guide [says](https://developers.openai.com/api/docs/guides/evaluation-best-practices) a dataset should include typical, edge and adversarial cases, and recommends mining your logs for good eval cases. Anthropic's docs list [edge cases](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) to cover: irrelevant or nonexistent input data, overly long input, poor or harmful user input, and ambiguous cases where humans would also disagree.

You do not need a large set to begin. Anthropic says [20 to 50 simple tasks drawn from real failures](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) is a great start, because early changes tend to have large effects that small samples can detect. Hamel Husain [uses an LLM](https://hamel.dev/blog/posts/evals/) to generate synthetic inputs, aiming for tests as challenging as possible while still representing how users interact. See [golden dataset](/learn/golden-dataset), [synthetic test data for evals](/learn/synthetic-test-data-for-evals) and [error analysis](/learn/error-analysis-llm).

## Handle non-determinism

The same input can give different outputs, so one pass tells you little. Anthropic [recommends running multiple trials](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) and reading two numbers. Pass@k is the chance of at least one correct result in k attempts, which suits tools where one success is enough. Pass^k is the chance that all k trials succeed, which suits customer-facing behavior that must be consistent. Anthropic's example is that a 75% per-trial success rate gives roughly 42% for pass^3.

Grade the result, not the route. Anthropic says it is [often better to grade what the agent produced, not the path it took](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), because rigid checks on exact steps are brittle.

## Set a pass bar and track it

Decide what counts as passing before you look at scores. Hamel Husain [says](https://hamel.dev/blog/posts/evals/) you do not necessarily need a 100% pass rate, because the rate is a product decision about which failures you will tolerate, and that you should track results over time to see whether you are making progress. Eugene Yan [advises](https://eugeneyan.com/writing/evals/) calibrating the bar to risk: stricter for external, high-stakes use, and for internal tools start small and iterate. OpenAI [suggests](https://developers.openai.com/api/docs/guides/evaluation-best-practices) a pass/fail threshold alongside numeric scores, and warns against "vibe-based evals" where quality is judged by whether it seems to work.

## Test safety separately

Quality tests do not cover abuse. Eugene Yan [recommends](https://eugeneyan.com/writing/evals/) adversarial prompts from public toxicity datasets, scored with a toxicity classifier, and notes that recent LLMs do a good job of keeping output harmless, so expect low failure rates in ordinary use. Anthropic's docs give a measurable [safety criterion](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests): fewer than 0.1% of 10,000 trials flagged for toxicity by a content filter. For attacks on the application itself, see [prompt injection](/learn/prompt-injection) and [LLM red teaming](/learn/llm-red-teaming).

## Keep testing in production

A fixed dataset is only as good as the inputs you thought of. OpenAI's guide [warns](https://developers.openai.com/api/docs/guides/evaluation-best-practices) against datasets that do not match production traffic, and tells teams to evaluate continuously instead of waiting until shipping. Real failures found in production become new test cases. Hamel Husain [says](https://hamel.dev/blog/posts/evals/) A/B testing is usually appropriate only for more mature products. See [online vs offline evaluation](/learn/online-vs-offline-evaluation).

## Running these tests in HoneyHive

HoneyHive's [experiments quickstart](https://docs.honeyhive.ai/v2/introduction/experiments-quickstart.md) shows the dataset layer: `evaluate()` runs a function over a test dataset, scores each output with an evaluator you write, and shows aggregated results in the Experiments dashboard. Unit tests for the code around the model stay in your own test runner.

## FAQ

### Can you unit test an LLM application?

Yes, for the code around the model. Pydantic AI's testing guide [swaps in a stand-in model](https://pydantic.dev/docs/ai/guides/testing/) to avoid the usage, latency and variability of real calls, and says such tests check your code, not answer quality. Answer quality is the job of evals.

### How do you test LLM output that changes every run?

Grade properties instead of exact strings, and run several trials. Anthropic [recommends multiple trials](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) and two metrics: pass@k (at least one of k succeeds) and pass^k (all k succeed).

### How many test cases do you need?

Start small. Anthropic says [20 to 50 simple tasks drawn from real failures](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) is a great start. See [golden dataset](/learn/golden-dataset) for how to build the set.

### Does an LLM test suite need a 100% pass rate?

Not necessarily. Hamel Husain [writes](https://hamel.dev/blog/posts/evals/) that the pass rate is a product decision that depends on the failures you are willing to tolerate. Anthropic expects [regression evals to pass at close to 100%](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), so the bar differs by test type.

### What is the difference between LLM testing and LLM evaluation?

Tests give repeatable pass or fail results, while evals score quality across many examples. In practice the model-facing tests are evals with a threshold. See [LLM evaluation](/learn/llm-evaluation).