How to detect LLM hallucinations
Detect LLM hallucinations by checking claims against source text, by sampling and comparing answers, or by human review. What each method needs and misses.
You detect a hallucination by testing a claim against something other than the model’s own confidence: a source document, a search result, other samples of the same model, or a person. Which test you can run depends on what you have. With the retrieved context, check each claim against it. With nothing but the prompt, compare several sampled answers. Either way, check claims one at a time, because a long answer is usually part right and part wrong.
For what hallucinations are and why they happen, see LLM hallucinations. This page covers finding them.
Pick the method by what you have
| Method | Needs | What it checks | Stated limit |
|---|---|---|---|
| Claim-by-claim check against context | The source text the model was given | Whether each statement is supported | Only as good as the checker; says nothing about facts outside the context |
| Claim-by-claim check with search | Search access, many model calls | Whether each statement is true in the world | A checker that disagrees with people on some share of facts |
| Sampling and consistency | Several samples per prompt, no sources | Whether the model gives the same answer each time | Misses errors the model repeats every time |
| Human review | Reviewers and a rubric | Anything | Slow and costly, so use it on samples and to check the automated methods |
Check claims against the source
When the app retrieves documents, the first question is whether the answer stays inside them. Es and colleagues define faithfulness for retrieval-augmented answers in two steps. An LLM splits the answer into statements, then checks each statement against the context for a yes or no. The score is the supported statements divided by all statements. A score of 1 means every claim traces to the context. It does not mean the claims are true, since the context can be wrong, and it does not reward a short answer that skips what the user needed. The paper keeps that separate, as answer relevance.
Min and colleagues built FActScore for the same reason: judging a long text as true or false hides the mix of correct and incorrect facts inside it. It reports the percentage of atomic facts that a reliable knowledge source supports. The authors say their automated estimator, which uses retrieval and a strong language model, has less than a 2% error rate against human evaluation, and that checking 6,500 generations from 13 models by hand would have cost $26K.
When there is no context to check against, Wei and colleagues built SAFE: an LLM splits a long answer into individual facts and checks each one with multi-step reasoning that includes Google Search queries. On about 16k facts it agreed with crowdsourced human annotators 72% of the time, and on a random sample of 100 disagreements it was right 76% of the time. The authors put it at more than 20 times cheaper than human annotators. That is cheap enough to run at scale, and a 28% disagreement rate is still a reason to audit a sample.
Compare samples when you have no source
Two methods use the model’s own variation as the signal. Manakul and colleagues propose SelfCheckGPT, which works without an external database and without access to token probabilities. If the model knows a fact, sampled answers tend to agree on it. For hallucinated facts, they say, sampled responses are likely to diverge and contradict one another.
Farquhar and colleagues published semantic entropy in Nature in June 2024. It samples several answers, groups them by meaning (two answers share a group if each entails the other), and computes entropy over the groups. Grouping by meaning keeps differently worded answers that say the same thing from counting as different. The paper targets confabulations, which it defines as claims that are both wrong and arbitrary, and it has a variant that works without output probabilities.
Both methods share a blind spot. The Nature authors state that semantic entropy does not help when a model’s outputs are systematically bad, and that it does not address a model that is confidently wrong. A model that repeats the same wrong date every time looks consistent. Sampling also multiplies cost: each check takes several calls per prompt before any judging.
Turn it into a number you can track
- Pick the unit. Per claim (FActScore, faithfulness) is more informative than per answer, which is a yes or no on a mixed output.
- Track abstentions apart from errors, as described in LLM hallucinations, so a model that refuses more does not look more accurate.
- Build a dataset of real questions with their source passages or reference answers. Golden datasets covers how.
- Label a sample by hand and measure how often the automated check agrees. Human evaluation and LLM judge limitations explain why this step is not optional. In the SAFE study, the checker and human annotators disagreed on 28% of facts, and only a hand review of those cases showed which side was right.
- Re-check on production traffic, because live questions drift from the dataset. See LLM monitoring.
Faithfulness checks pair with LLM-as-a-judge prompts and with the retrieval side of a RAG pipeline, since a wrong answer may come from retrieval that never fetched the fact.
In HoneyHive
HoneyHive’s evaluator templates include an Answer Faithfulness Evaluator, which checks whether an answer sticks to the supplied context, and a Context Relevance Evaluator for what the retriever returned. Online evaluations run evaluators asynchronously on ingested traces, so they add no latency, and a sampling percentage controls LLM-judge cost. Alerts can fire on an evaluator score, and annotation queues route traces to reviewers whose labels you can use to check the evaluator.
FAQ
How do you detect hallucinations in an LLM?
Split the output into individual claims and check each against a trusted source, or sample the model several times and flag answers that disagree with each other. Which one you can use depends on whether you have source text and how many calls you can afford.
What is a hallucination rate?
Usually the share of claims that are unsupported. FActScore reports the percentage of atomic facts supported by a reliable knowledge source, and the faithfulness score defined by Es and colleagues is the number of supported statements divided by all statements.
Can you detect hallucinations without a reference answer or source?
Partly. SelfCheckGPT and semantic entropy compare several sampled answers and need no external database, but they miss errors a model makes consistently.
Can an LLM judge detect hallucinations reliably?
Not without checking it. In the SAFE study an LLM-based checker agreed with crowdsourced human annotators on 72% of about 16k facts and was right in 76% of a sample of 100 disagreements, so it can match or beat crowd labels, but you only learn how it does on your data by reviewing a sample of its verdicts. See LLM judge limitations.
How do you detect hallucinations in production?
Run a cheap check on every trace and an LLM-based check on a sampled share, alert when the score drops, and send low scores to human reviewers. HoneyHive online evaluations work this way.