# Data drift vs concept drift in LLM apps

> Data drift is a change in what goes into your LLM app; concept drift is a change in what a correct answer is.

Source: https://www.honeyhive.ai/learn/data-drift-vs-concept-drift (updated 2026-11-03)

Data drift means the inputs to your system changed. Concept drift means the relationship between an input and its correct output changed. In classic ML both are measured against a model you trained. In an LLM app you usually did not train the model, so the picture is messier: the inputs, the retrieved context and the model behind an API can each move, and what counts as a correct answer can move too.

## The two definitions

A widely cited 2014 survey of concept drift in ACM Computing Surveys describes it in its abstract as the case where [the relation between the input data and the target variable changes over time](https://eprints.bournemouth.ac.uk/22491/). Data drift is the other half: the distribution of the inputs changes, and the input-to-answer rule may or may not.

The difference matters because the checks differ. You can see data drift by comparing today's inputs with a baseline. You cannot see concept drift that way, because the inputs can look the same while the correct answer is different. You need some measure of the answer, which for an LLM app means a quality score on traces. See [LLM evaluation](/learn/llm-evaluation) for how those scores are produced.

## Where each shows up in an LLM app

LLM apps add a third source of change that neither term was written for: the model itself. A study of hosted models found that the March and June 2023 versions of GPT-3.5 and GPT-4 [performed and behaved very differently on seven task types](https://arxiv.org/abs/2307.09009), including GPT-4's accuracy at telling prime from composite numbers falling from 84% to 51%. The authors conclude that LLM services need continuous monitoring. Providers also retire models; OpenAI's deprecations page says it [regularly retires older models](https://developers.openai.com/api/docs/deprecations) and tells users to migrate before the shutdown date.

The table maps each kind of change to the term that fits best. The mapping is a working guide, not a standard definition.

| What changed | Closest term | What you see | Check that catches it |
|---|---|---|---|
| Users ask about new topics, in new phrasing or languages | Data drift | Inputs no longer match your test set | Compare input features (length, language, topic, intent) with a baseline period |
| Retrieved documents or tool responses change | Data drift in the context | Same question, different context sent to the model | Compare retrieved content and tool outputs across periods; check groundedness scores |
| A policy, price or fact changes, so the right answer changes | Concept drift | Inputs look the same; answers that used to be right are now wrong | Evaluator scores or human review on sampled traces against current ground truth |
| What users or reviewers count as good changes | Concept drift (the label moved) | Scores fall without any change in inputs or model | Re-read a sample and update the rubric |
| The provider changes or retires the model | Neither, strictly | Quality or format shifts with no change in inputs or ground truth | Run your fixed test set against the new model before switching; watch scores after |
| You change the prompt | Not drift | A step change at a known time | Compare scores before and after the release |

Real incidents often mix several rows. Look at inputs first because it is cheap, then at scores to see whether quality moved.

## How to monitor for each

Input checks are cheap and need no labels, so run them on all traffic. Quality checks are what show concept drift and model changes, but they cost more. Code checks can run on every trace; model-graded and human checks usually run on a sample.

1. Keep a frozen test set with known good answers and rerun it whenever the model, the prompt or the knowledge changes. This is how you tell a model change from real-world change.
2. Score a sample of live traces and chart the scores over time, split by input type, so a drop points to a segment.
3. Review a few traces by hand each week. A rubric can go stale without the scores moving, which is the label-side form of concept drift.
4. When a score drops, find out which row of the table applies before changing anything. Retraining is rarely the answer for an app that calls a hosted model; changing the prompt, the documents, the test set or the model version usually is.

[LLM observability](/learn/llm-observability) covers the traces these checks run on.

## Running the quality checks in HoneyHive

HoneyHive's [online evaluations](https://docs.honeyhive.ai/v2/monitoring/onlineevals.md) run Python or LLM evaluators asynchronously on ingested traces without adding latency to the app. Filters pick which traces an evaluator runs on, and a sampling rate limits cost. Scores can be charted over time and used for alerts, and each evaluated trace shows its scores next to its inputs and outputs. That covers the "score a sample and chart it" step above.

## FAQ

### Is model drift the same as concept drift?

No. "Model drift" is a loose label for any drop in a deployed model's quality. Concept drift is one specific cause: the relation between inputs and the correct output changed. Quality can also fall because the inputs changed (data drift) or, with a hosted LLM, because the provider replaced or updated the model.

### Can you detect concept drift without labels?

Not reliably. Concept drift is a change in the relation between input and target, so you need some measure of the target. For an LLM app that means evaluator scores or human review on a sample of production traces. A check on input distributions alone can stay flat while the correct answer changes.

### Do LLM apps need retraining when they drift?

Often not, because most teams do not train the model. The fix is usually a change to the prompt, the retrieved documents, the test set or the evaluator, or a move to a newer model version, followed by a re-run of the evaluation.