How to test LLM applications

To test an LLM application, split it into parts and test each the way it behaves. Unit-test the code around the model (prompt building, parsing, validation) with a stub in place of the model. Grade real model outputs against a fixed dataset with graders that tolerate variation. Then check production traffic for what the dataset missed. Hamel Husain describes three levels of this kind (unit tests, human and model evaluation, A/B tests) and says each costs more than the one before it, which sets how often to run it.

This page is the overall approach. For the CI gate that reruns a dataset on every change, see LLM regression testing. For agents, see how to test AI agents. For metrics and judges, see LLM evaluation.

What to test at each layer

Layer	What it checks	How to grade	When to run
Code around the model	Prompt templates, parsing, validation, error handling	Ordinary assertions with the model stubbed	Every commit
Model behavior	Quality on typical, edge and adversarial inputs	Code-based, model-based or human graders over a fixed dataset	On every prompt, model or code change
Safety and abuse	Harmful, off-topic or manipulated inputs	Adversarial prompts and a content classifier or judge	Before release and on a schedule
Production	Real user inputs the dataset never covered	Sampled traces, user feedback, A/B tests	Continuously

Test the code around the model without the model

Most of an LLM application is ordinary code, and it should have ordinary tests. Pydantic AI’s testing guide recommends replacing the model with a stand-in so tests avoid the usage, latency and variability of real LLM calls. Its TestModel is described as the simplest and fastest way to exercise most of your application code, and FunctionModel lets you write your own response logic. Setting ALLOW_MODEL_REQUESTS to False guards against accidental real requests during tests.

The guide is explicit that these unit tests check your own code, not answer quality. Other frameworks have their own stand-ins, and the idea transfers: a test that needs a live model call is an eval, and it belongs in the next layer. If your application depends on a fixed output shape, test that the parser handles malformed output as well as valid output; see structured output.

Hamel Husain describes the assertions he uses at this level as checks “like you would write in pytest” that run fast and cheaply. He scopes them by feature and scenario, such as one match, several matches and no matches, and adds generic checks like a regex that catches exposed UUIDs in output.

Grade real outputs on a fixed dataset

For the model’s own behavior, run the application on a fixed set of inputs and grade each output. Anthropic’s guide compares three kinds of grader.

Grader	Strengths	Weaknesses
Code-based (string match, schema checks, static analysis)	Fast, cheap, objective, reproducible, easy to debug	Can reject valid answers that differ from the expected pattern; limited for some subjective tasks
Model-based (rubric scoring, pairwise comparison)	Flexible, scales, handles open-ended output	Results vary between runs, costs more, needs calibration against humans
Human (expert review, spot checks)	The gold standard, calibrates the model graders	Expensive and slow

Use the cheapest grader that can tell a good output from a bad one. Anthropic’s docs advise automating grading where possible and say more questions with slightly lower-signal automated grading beat fewer hand-graded ones. Where you use a model as the grader, the same docs say it is generally best practice to use a different model from the one that produced the output, and OpenAI advises scaling up an LLM judge only once it consistently agrees with human annotations. See LLM as a judge.

Pick metrics that fit the task. Eugene Yan argues that accuracy is too coarse for classification and extraction and that recall and precision tell you more, and that off-the-shelf evals barely correlate with application-specific performance. OpenAI lists overly generic metrics such as perplexity or BLEU as a pitfall. Define success criteria first. Anthropic’s docs say criteria should be specific, measurable, achievable and relevant. For metric choices see LLM evaluation metrics.

Build the dataset from real failures

OpenAI’s guide says a dataset should include typical, edge and adversarial cases, and recommends mining your logs for good eval cases. Anthropic’s docs list edge cases to cover: irrelevant or nonexistent input data, overly long input, poor or harmful user input, and ambiguous cases where humans would also disagree.

You do not need a large set to begin. Anthropic says 20 to 50 simple tasks drawn from real failures is a great start, because early changes tend to have large effects that small samples can detect. Hamel Husain uses an LLM to generate synthetic inputs, aiming for tests as challenging as possible while still representing how users interact. See golden dataset, synthetic test data for evals and error analysis.

Handle non-determinism

The same input can give different outputs, so one pass tells you little. Anthropic recommends running multiple trials and reading two numbers. Pass@k is the chance of at least one correct result in k attempts, which suits tools where one success is enough. Pass^k is the chance that all k trials succeed, which suits customer-facing behavior that must be consistent. Anthropic’s example is that a 75% per-trial success rate gives roughly 42% for pass^3.

Grade the result, not the route. Anthropic says it is often better to grade what the agent produced, not the path it took, because rigid checks on exact steps are brittle.

Set a pass bar and track it

Decide what counts as passing before you look at scores. Hamel Husain says you do not necessarily need a 100% pass rate, because the rate is a product decision about which failures you will tolerate, and that you should track results over time to see whether you are making progress. Eugene Yan advises calibrating the bar to risk: stricter for external, high-stakes use, and for internal tools start small and iterate. OpenAI suggests a pass/fail threshold alongside numeric scores, and warns against “vibe-based evals” where quality is judged by whether it seems to work.

Test safety separately

Quality tests do not cover abuse. Eugene Yan recommends adversarial prompts from public toxicity datasets, scored with a toxicity classifier, and notes that recent LLMs do a good job of keeping output harmless, so expect low failure rates in ordinary use. Anthropic’s docs give a measurable safety criterion: fewer than 0.1% of 10,000 trials flagged for toxicity by a content filter. For attacks on the application itself, see prompt injection and LLM red teaming.

Keep testing in production

A fixed dataset is only as good as the inputs you thought of. OpenAI’s guide warns against datasets that do not match production traffic, and tells teams to evaluate continuously instead of waiting until shipping. Real failures found in production become new test cases. Hamel Husain says A/B testing is usually appropriate only for more mature products. See online vs offline evaluation.

Running these tests in HoneyHive

HoneyHive’s experiments quickstart shows the dataset layer: evaluate() runs a function over a test dataset, scores each output with an evaluator you write, and shows aggregated results in the Experiments dashboard. Unit tests for the code around the model stay in your own test runner.

FAQ

Can you unit test an LLM application?

Yes, for the code around the model. Pydantic AI's testing guide swaps in a stand-in model to avoid the usage, latency and variability of real calls, and says such tests check your code, not answer quality. Answer quality is the job of evals.

How do you test LLM output that changes every run?

Grade properties instead of exact strings, and run several trials. Anthropic recommends multiple trials and two metrics: pass@k (at least one of k succeeds) and pass^k (all k succeed).

How many test cases do you need?

Start small. Anthropic says 20 to 50 simple tasks drawn from real failures is a great start. See golden dataset for how to build the set.

Does an LLM test suite need a 100% pass rate?

Not necessarily. Hamel Husain writes that the pass rate is a product decision that depends on the failures you are willing to tolerate. Anthropic expects regression evals to pass at close to 100%, so the bar differs by test type.

What is the difference between LLM testing and LLM evaluation?

Tests give repeatable pass or fail results, while evals score quality across many examples. In practice the model-facing tests are evals with a threshold. See LLM evaluation.