How to reduce hallucinations in LLM apps
Reduce LLM app hallucinations by fixing the layer that causes them: retrieval, context order, citations, abstention or verification.
Hallucinations in an app come from different layers, and each layer has its own fix. If the model never saw the fact, retrieval is the problem. If it saw the fact and ignored it, the problem is context or generation. If it answered when it should have declined, the problem is how the app handles uncertainty. Swapping in a bigger model or lowering temperature treats all of these the same way, which is why it often disappoints.
Anthropic’s prompt techniques (allow “I don’t know”, quote first, restrict to provided documents) are in LLM hallucinations. This page covers the system around the prompt. To find the layer first, see detecting hallucinations.
Match the symptom to the layer
| Symptom | Likely layer | What to change | Evidence |
|---|---|---|---|
| Wrong or invented facts, no source involved | Knowledge | Put the source in the prompt with retrieval | RAG output was more factual than a parametric-only baseline |
| Right passage retrieved, answer ignores it | Context | Move key passages to the start or end, shorten the context | Accuracy drops for information in the middle |
| Answer cannot be traced to a document | Generation | Require citations tied to the source text | Anthropic’s citations feature returns exact passages |
| Confident answer where retrieval found nothing | Uncertainty | Retrieve on demand, allow abstention, reward it in evals | Self-RAG, and the scoring argument in Kalai and colleagues |
| Long answer with a few wrong facts | Verification | Draft, then check each claim and rewrite | Chain-of-Verification reports fewer hallucinations |
| Retrieved passages are off topic | Retrieval | Fix chunking and ranking, then re-measure | Context relevance is a measurable score |
Retrieval
Lewis and colleagues paired a generator with a dense index of Wikipedia, and reported that their models produce more specific, diverse and factual language than a parametric-only baseline. The same abstract lists provenance and updating knowledge as open problems for pretrained models, which retrieval is meant to help with. It is not a guarantee. Retrieval adds a second source of error: the wrong passages. Es and colleagues define context relevance as the share of retrieved sentences needed to answer the question, so you can measure whether retrieval is the problem. Chunking strategies and agentic RAG cover the fixes.
Context order and size
Liu and colleagues found that model performance is often highest when relevant information is at the start or end of the input and degrades significantly when it is in the middle, including in models built for long contexts. Pasting retrieved passages in rank order is not neutral. Retrieve fewer, put the best first or last, and see context rot and context engineering.
Citations
A citation makes a claim checkable by the reader and by your code. Anthropic’s citations feature returns the exact passages behind each claim. Anthropic says the API parses them into pointers that are guaranteed to be valid for the provided documents, that cited_text does not count toward output tokens, and that in its own evaluations the feature is significantly more likely to cite the most relevant quotes than prompt-based approaches. A valid pointer shows where the model says it got a claim. It does not show that the passage supports the claim, so keep a faithfulness check on top.
Retrieving and answering only when needed
Standard RAG retrieves a fixed number of passages whether or not they help. Asai and colleagues trained Self-RAG to retrieve on demand and to critique both the passages and its own output with reflection tokens. They report gains in factuality and citation accuracy on long-form generation over ChatGPT and retrieval-augmented Llama2-chat. It requires training a model, so a simpler version of the idea is routing in the app: skip retrieval for questions that do not need it, and return “I don’t know” when retrieval finds nothing relevant.
Abstention also depends on how you score. Kalai and colleagues argue that training and evaluation reward guessing over admitting uncertainty. If your own eval gives no credit for declining, tuning against it pushes the app to guess.
Verify before returning
Dhuliawala and colleagues describe Chain-of-Verification in four steps: the model drafts a response, plans verification questions, answers them independently so the answers are not biased by the draft, and writes a final verified response. They report fewer hallucinations on list questions from Wikidata, closed-book MultiSpanQA and long-form generation. It takes several model calls per answer, so use it where a wrong claim costs more than the latency. For checks that block output in the app, see AI guardrails.
Prove the change worked
Every fix above can also add a new failure, so measure.
- Fix a dataset of real questions with source passages, as in golden datasets.
- Run the old and new versions on it and compare a per-claim score such as faithfulness, with abstentions counted separately.
- Read the failures by hand. Error analysis tells you which layer each one came from.
- Gate releases on the score, then watch it on production traces.
HoneyHive’s CI regression detection guide sets up a GitHub Actions workflow that runs an evaluation on each pull request and compares every metric with a baseline run from the base commit. If any metric degrades on at least one datapoint, the check fails and blocks the merge. The evaluator templates include an Answer Faithfulness Evaluator and a Context Relevance Evaluator, and alerts can fire on an evaluator score in production.
FAQ
How do you reduce hallucinations in an LLM app?
Find which layer the errors come from, then change that layer: retrieve better passages, put key evidence where the model uses it, require citations, let the model abstain, or verify a draft before returning it. Test each change on the same dataset.
Does RAG stop hallucinations?
No. The original RAG paper reports more factual output than a parametric-only baseline, not an absence of errors. The model can still ignore or misuse what was retrieved. See RAG pipelines.
Where should relevant information go in the prompt?
Near the start or the end. Liu and colleagues found performance is often highest there and drops when the relevant information sits in the middle of a long context.
What is Chain-of-Verification?
A method from Dhuliawala and colleagues in which the model drafts an answer, plans verification questions, answers them independently, then writes a final verified response.
How do you know a change reduced hallucinations?
Run the old and new versions on the same dataset and compare a per-claim score such as faithfulness, with abstentions counted separately. Then watch the same score on production traces.