# LLM regression testing in CI

> LLM regression testing reruns a fixed eval set on every prompt, model or code change in CI and fails the build when scores drop against a baseline.

Source: https://www.honeyhive.ai/learn/llm-regression-testing (updated 2026-11-11)

LLM regression testing is rerunning a fixed set of eval tasks every time a prompt, model or piece of code changes, and failing the build when quality drops. Anthropic describes regression evals as the check that answers ["Does the agent still handle all the tasks it used to?"](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), and says they should pass at a nearly 100% rate. In CI, the suite runs on each pull request and compares the result with a baseline from the base branch.

This page covers the CI gate. For test layers around the model, see [how to test AI agents](/learn/testing-ai-agents). For metrics and judges, see [LLM evaluation](/learn/llm-evaluation) and [LLM as a judge](/learn/llm-as-a-judge).

## Why LLM apps regress

Three things change without a bug in your code.

- **Your own edits.** A prompt tweak that fixes one case can break others. OpenAI's guide [recommends](https://developers.openai.com/api/docs/guides/evaluation-best-practices) setting up continuous evaluation "to run evals on every change."
- **The provider's model.** Chen, Zaharia and Zou [compared](https://arxiv.org/abs/2307.09009) March and June 2023 versions of GPT-3.5 and GPT-4. GPT-4's accuracy at telling prime from composite numbers fell from 84% to 51%, and both models made more formatting errors in code generation. They concluded that the behavior of the "same" LLM service can change substantially in a short time.
- **Forced model upgrades.** Anthropic [retires older models](https://platform.claude.com/docs/en/about-claude/model-deprecations) on a schedule and gives [at least 60 days' notice](https://platform.claude.com/docs/en/about-claude/model-deprecations) for publicly released ones. Its docs advise testing your application on the replacement well before the retirement date. A parameter can also be removed: `temperature`, `top_p` and `top_k` [return a 400 error](https://platform.claude.com/docs/en/about-claude/model-deprecations) from API version 2026-10-21, so an upgrade can break requests as well as quality.

## What a regression suite contains

Anthropic splits evals in two. Capability evals [start at a low pass rate](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) and measure what the system can do. Regression evals protect against backsliding, where [a decline in score signals that something is broken](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents). Capability evals with high pass rates can graduate into the regression suite and run continuously to catch drift.

| Question | Guidance from the sources |
|---|---|
| How big is the first suite? | [20 to 50 simple tasks drawn from real failures](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) is a good start, and teams should not wait for hundreds. |
| Where do cases come from? | Real failures. Turn each bug or bad trace into a case; see [golden dataset](/learn/golden-dataset) and [error analysis](/learn/error-analysis-llm). |
| What kind of checks? | Fast assertions first. Hamel Husain [writes](https://hamel.dev/blog/posts/evals/) that unit tests for LLMs are assertions like you would write in pytest, and you can run them every time your code changes. Add model-graded checks where assertions cannot reach; see [LLM judge limitations](/learn/llm-judge-limitations). |
| How do you check the graders? | Anthropic says you will not know if graders work [unless you read transcripts and grades](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) from many trials. |

## Handling non-deterministic output

OpenAI's guide opens by [noting](https://developers.openai.com/api/docs/guides/evaluation-best-practices) that models sometimes produce different output from the same input, so an exact-match assert on free text will flake. Compare scores instead.

Anthropic runs multiple trials per task and offers two ways to read them. pass@k is [the likelihood of at least one correct solution in k attempts](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents). pass^k is [the probability that all k trials succeed](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents). For a regression gate, pass^k is the stricter fit, because a flaky pass is not a pass. The cost is more model calls per run, so keep the per-commit suite small.

## Wiring it into CI

The mechanics come from GitHub Actions, and other CI systems have equivalents.

| Need | How GitHub Actions handles it |
|---|---|
| Run on every pull request | The [`pull_request` event](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows) runs a workflow on activity on a pull request. |
| Skip when nothing relevant changed | A [`paths` filter](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows) runs the workflow only when a pull request changes specific files, for example prompts or agent code. |
| Run a larger suite nightly | The [`schedule` event](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows) triggers at a set time, no more often than every 5 minutes, and only on the latest commit of the default branch. |
| Use a model API key | [Secrets are not passed](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows) to the runner when a workflow is triggered from a forked repository, so fork pull requests cannot run live evals. |

One way to split the work is two tiers: a small suite on each pull request that blocks the merge, and a larger scheduled suite that catches provider drift while no one is pushing code. Anthropic's guidance supports both: automated evals suit [CI/CD on each change and model upgrade](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), and a regression suite is run continuously to catch drift.

### What to compare against

Absolute thresholds go stale. Comparing against a baseline from the base commit tells you whether this change made things worse. Decide up front whether any drop fails the build or only drops on critical metrics, and how a deliberate change to the expected answers is approved.

## Running it in HoneyHive

HoneyHive's [CI regression detection](https://docs.honeyhive.ai/v2/evaluation/ci-regression-detection) uses this baseline pattern with GitHub Actions. On each push to `main` a script runs `evaluate()` and stores the run under an ID derived from the commit SHA. On each pull request the same script runs on the head commit, then `compare_runs()` compares it with the base-commit run. The job [exits 1 if any metric degraded](https://docs.honeyhive.ai/v2/evaluation/ci-regression-detection), and a pull request comment shows the changes either way. By default any degraded metric fails the check; the docs show filtering to a set of critical metrics instead. Teams outside Python can use the run comparison REST endpoint. To build the dataset from production traces, see the [dataset curation docs](https://docs.honeyhive.ai/v2/datasets/dataset-curation).

## FAQ

### What is LLM regression testing?

It is rerunning a fixed set of eval tasks after every change to check the application still handles what it used to. Anthropic calls these [regression evals](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) and says they should have a nearly 100% pass rate.

### How do you regression test non-deterministic output?

Run several trials per task and compare aggregate scores, not exact strings. Anthropic [runs multiple trials](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) because model outputs vary between runs.

### How many test cases do you need to start?

Anthropic says [20 to 50 simple tasks drawn from real failures](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) is a great start. See [golden dataset](/learn/golden-dataset) for building the set.

### Should regression tests run on every pull request?

Run the cheap ones on every change and the costly ones on a schedule. Anthropic says automated evals can [run on every commit](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), while GitHub Actions can also trigger a workflow on a [schedule](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows).

### Why do LLM apps regress when the code did not change?

The model can change under you. A [2023 study](https://arxiv.org/abs/2307.09009) found GPT-4's accuracy at spotting prime numbers fell from 84% to 51% between March and June versions, and providers [retire models](https://platform.claude.com/docs/en/about-claude/model-deprecations) on a schedule.