LLM regression testing in CI
LLM regression testing reruns a fixed eval set on every prompt, model or code change in CI and fails the build when scores drop against a baseline.
LLM regression testing is rerunning a fixed set of eval tasks every time a prompt, model or piece of code changes, and failing the build when quality drops. Anthropic describes regression evals as the check that answers “Does the agent still handle all the tasks it used to?”, and says they should pass at a nearly 100% rate. In CI, the suite runs on each pull request and compares the result with a baseline from the base branch.
This page covers the CI gate. For test layers around the model, see how to test AI agents. For metrics and judges, see LLM evaluation and LLM as a judge.
Why LLM apps regress
Three things change without a bug in your code.
- Your own edits. A prompt tweak that fixes one case can break others. OpenAI’s guide recommends setting up continuous evaluation “to run evals on every change.”
- The provider’s model. Chen, Zaharia and Zou compared March and June 2023 versions of GPT-3.5 and GPT-4. GPT-4’s accuracy at telling prime from composite numbers fell from 84% to 51%, and both models made more formatting errors in code generation. They concluded that the behavior of the “same” LLM service can change substantially in a short time.
- Forced model upgrades. Anthropic retires older models on a schedule and gives at least 60 days’ notice for publicly released ones. Its docs advise testing your application on the replacement well before the retirement date. A parameter can also be removed:
temperature,top_pandtop_kreturn a 400 error from API version 2026-10-21, so an upgrade can break requests as well as quality.
What a regression suite contains
Anthropic splits evals in two. Capability evals start at a low pass rate and measure what the system can do. Regression evals protect against backsliding, where a decline in score signals that something is broken. Capability evals with high pass rates can graduate into the regression suite and run continuously to catch drift.
| Question | Guidance from the sources |
|---|---|
| How big is the first suite? | 20 to 50 simple tasks drawn from real failures is a good start, and teams should not wait for hundreds. |
| Where do cases come from? | Real failures. Turn each bug or bad trace into a case; see golden dataset and error analysis. |
| What kind of checks? | Fast assertions first. Hamel Husain writes that unit tests for LLMs are assertions like you would write in pytest, and you can run them every time your code changes. Add model-graded checks where assertions cannot reach; see LLM judge limitations. |
| How do you check the graders? | Anthropic says you will not know if graders work unless you read transcripts and grades from many trials. |
Handling non-deterministic output
OpenAI’s guide opens by noting that models sometimes produce different output from the same input, so an exact-match assert on free text will flake. Compare scores instead.
Anthropic runs multiple trials per task and offers two ways to read them. pass@k is the likelihood of at least one correct solution in k attempts. pass^k is the probability that all k trials succeed. For a regression gate, pass^k is the stricter fit, because a flaky pass is not a pass. The cost is more model calls per run, so keep the per-commit suite small.
Wiring it into CI
The mechanics come from GitHub Actions, and other CI systems have equivalents.
| Need | How GitHub Actions handles it |
|---|---|
| Run on every pull request | The pull_request event runs a workflow on activity on a pull request. |
| Skip when nothing relevant changed | A paths filter runs the workflow only when a pull request changes specific files, for example prompts or agent code. |
| Run a larger suite nightly | The schedule event triggers at a set time, no more often than every 5 minutes, and only on the latest commit of the default branch. |
| Use a model API key | Secrets are not passed to the runner when a workflow is triggered from a forked repository, so fork pull requests cannot run live evals. |
One way to split the work is two tiers: a small suite on each pull request that blocks the merge, and a larger scheduled suite that catches provider drift while no one is pushing code. Anthropic’s guidance supports both: automated evals suit CI/CD on each change and model upgrade, and a regression suite is run continuously to catch drift.
What to compare against
Absolute thresholds go stale. Comparing against a baseline from the base commit tells you whether this change made things worse. Decide up front whether any drop fails the build or only drops on critical metrics, and how a deliberate change to the expected answers is approved.
Running it in HoneyHive
HoneyHive’s CI regression detection uses this baseline pattern with GitHub Actions. On each push to main a script runs evaluate() and stores the run under an ID derived from the commit SHA. On each pull request the same script runs on the head commit, then compare_runs() compares it with the base-commit run. The job exits 1 if any metric degraded, and a pull request comment shows the changes either way. By default any degraded metric fails the check; the docs show filtering to a set of critical metrics instead. Teams outside Python can use the run comparison REST endpoint. To build the dataset from production traces, see the dataset curation docs.
FAQ
What is LLM regression testing?
It is rerunning a fixed set of eval tasks after every change to check the application still handles what it used to. Anthropic calls these regression evals and says they should have a nearly 100% pass rate.
How do you regression test non-deterministic output?
Run several trials per task and compare aggregate scores, not exact strings. Anthropic runs multiple trials because model outputs vary between runs.
How many test cases do you need to start?
Anthropic says 20 to 50 simple tasks drawn from real failures is a great start. See golden dataset for building the set.
Should regression tests run on every pull request?
Run the cheap ones on every change and the costly ones on a schedule. Anthropic says automated evals can run on every commit, while GitHub Actions can also trigger a workflow on a schedule.
Why do LLM apps regress when the code did not change?
The model can change under you. A 2023 study found GPT-4's accuracy at spotting prime numbers fell from 84% to 51% between March and June versions, and providers retire models on a schedule.