# How to evaluate RAG systems

> Evaluate RAG in two parts: did retrieval find the right passages, and did the model answer from them. Metrics, test sets and fixes for each.

Source: https://www.honeyhive.ai/learn/rag-evaluation (updated 2026-11-11)

To evaluate a RAG system, score its two halves separately and then score the final answer. Retrieval is judged on whether the right passages came back. Generation is judged on whether the answer is faithful to those passages and actually answers the question. A [survey of RAG evaluation](https://arxiv.org/abs/2504.14891) reduces the system to those two problems: "the retrieval of the ground truth, and the generation of the response that closely aligns with the gold answer."

Scoring only the final answer hides which half failed. The same survey notes that many components are still evaluated with end-to-end metrics, "with a lack of comprehensive functional decomposition evaluation."

## The three things to score

Es and colleagues, in the [paper behind a reference-free RAG metric suite](https://arxiv.org/abs/2309.15217), name three dimensions: whether retrieval finds relevant, focused context passages, whether the model uses those passages faithfully, and the quality of the generation itself. [ARES](https://arxiv.org/abs/2311.09476) scores a similar set: context relevance, answer faithfulness and answer relevance.

| Dimension | Question it answers | Needs |
|---|---|---|
| Retrieval quality | Did the right passages come back, and high enough in the ranking? | Question plus the passages that should be found |
| Faithfulness (groundedness) | Does the answer stick to what was retrieved? | Question, retrieved context, answer |
| Answer quality | Is the answer correct and does it address the question? | Question, answer, and usually a reference answer |

Google Cloud's [guide](https://cloud.google.com/blog/products/ai-machine-learning/optimizing-rag-retrieval) defines response groundedness as "the extent to which the generated text aligns with the factual information retrieved from the source documents." A faithful answer can still be wrong if retrieval fetched the wrong passages, which is why the dimensions are scored apart. See [hallucination detection](/learn/hallucination-detection) for the faithfulness side.

## Retrieval metrics

The survey defines the standard ranking metrics:

- Recall@K: "the portion of relevant instances that have been retrieved over the total amount of relevant cases."
- Precision@K: "the fraction of relevant instances among the retrieved instances."
- MRR (mean reciprocal rank): "the average of the reciprocal ranks of the first correct answer for a set of queries."
- NDCG: accounts "for the position of the relevant documents."

Anthropic's [contextual retrieval post](https://www.anthropic.com/news/contextual-retrieval) shows one in practice. It reports 1 minus recall@20, "the percentage of relevant documents that fail to be retrieved within the top 20 chunks," and measured a 5.7% failure rate for a baseline setup, 3.7% with contextual embeddings and 1.9% after adding BM25 and reranking. Those numbers describe Anthropic's test data, not yours.

These metrics need labels saying which chunks are relevant for each question. If you do not have them, use an LLM judge for context relevance instead (see below).

## Generation metrics

The survey lists reference-based text metrics for generation: exact match, [ROUGE](/learn/rouge-score), [BLEU](/learn/bleu-score), [BERTScore](/learn/bertscore), METEOR and [perplexity](/learn/perplexity-metric). It adds that these "are not directly equivalent to retrieval and generation assessment methods," because in RAG the two operations alternate.

A model can also judge faithfulness and correctness. Google Cloud's [guide](https://cloud.google.com/blog/products/ai-machine-learning/optimizing-rag-retrieval) describes pointwise judging (a model assigns a score) and pairwise judging (a model picks the better of two responses), next to computation-based metrics such as ROUGE and BLEU. The Hugging Face [cookbook](https://huggingface.co/learn/cookbook/en/rag_evaluation) has a judge compare each answer to a reference answer on a 1 to 5 rubric, writing feedback before the score. Read [LLM-as-a-judge](/learn/llm-as-a-judge) for how to build one and [judge limitations](/learn/llm-judge-limitations) for how it misleads.

## Build the test set

Google Cloud's guide says the test set should cover "a broad subset of the underlying data" and include "variations in phrasing and question complexity that match real-world use cases." It suggests golden questions that are simple, complex, multi-part and misspelled, with reference answers so more metrics become possible. A reviewed set of this kind is a [golden dataset](/learn/golden-dataset).

If you have no questions yet, you can generate them. The Hugging Face cookbook has an LLM write question-answer pairs from knowledge-base chunks, then has critique agents score each question on groundedness, relevance and stand-alone quality, keeping only questions where all three score at least 4. It recommends more than 200 pairs for a real knowledge base because about half are filtered out. More on this in [synthetic test data for evals](/learn/synthetic-test-data-for-evals).

Judges and synthetic data both drift from human judgment. [ARES](https://arxiv.org/abs/2311.09476) addresses this with a small set of human annotations and prediction-powered inference to correct judge errors, and reports staying accurate with only a few hundred human labels. Google Cloud [recommends](https://cloud.google.com/blog/products/ai-machine-learning/optimizing-rag-retrieval) human evaluation after automated testing has made baseline quality solid, because automated metrics miss "the tone of responses, the clarity of explanations, and potential ambiguity." See [human evaluation](/learn/human-evaluation-llm).

## Find which half is failing

Google Cloud's guide says RAG "is fundamentally based on two components": retrieval accuracy and the context passed to the model. Its method is to run a baseline, change one component, rerun, and read the score difference as the effect of that change. Keep the questions, reference answers and settings fixed, so that "a change in evaluation scores is attributable to a single feature alone."

| Symptom | Likely half | What to vary |
|---|---|---|
| Relevant passage missing from the top K | Retrieval | Embedding model, number of chunks returned, [chunk size and overlap](/learn/chunking-strategies), metadata, reranking |
| Right passage retrieved, answer ignores or contradicts it | Generation | Prompt, model, how much context is passed |
| Answer is faithful but off-topic or incomplete | Either | Check retrieval recall first, then the prompt |

Google Cloud lists the same levers: number of neighbors, embedding model, chunk size and overlap, summarizing long chunks and adding metadata on the retrieval side, and model, prompt and context choice on the generation side. The Hugging Face [cookbook](https://huggingface.co/learn/cookbook/en/rag_evaluation) lists chunk size, embedding model, reranking and the reader model as settings to tune, and calls chunk size easy to tune and very impactful. The [RAG pipeline](/learn/rag-pipeline) page describes the stages these levers belong to.

## Run it on real traffic too

An offline test set only covers the questions you thought of. Google Cloud's authors [warn](https://cloud.google.com/blog/products/ai-machine-learning/optimizing-rag-retrieval) that RAG systems that are not thoroughly evaluated lead to "silent failures." Keep a way to inspect what was retrieved for live queries and to add failures to the test set. [LLM evaluation](/learn/llm-evaluation) covers offline and online evaluation in general, and [LLM observability](/learn/llm-observability) covers capturing the retrieval step.

## Evaluating RAG in HoneyHive

HoneyHive's [LlamaIndex integration](https://docs.honeyhive.ai/v2/integrations/llamaindex.md) traces query engines, retrieval and embeddings, so retrieved context is visible per query. Its [evaluator templates](https://docs.honeyhive.ai/v2/evaluators/evaluator-templates.md) include Answer Faithfulness, Answer Relevance, Context Relevance and Context Coverage. You can [curate datasets from production traces](https://docs.honeyhive.ai/v2/datasets/dataset-curation.md) and run [experiments](https://docs.honeyhive.ai/v2/evaluation/introduction.md) on them to compare a pipeline before and after a change.

## FAQ

### How do you evaluate a RAG system?

Score retrieval and generation separately, then the final answer. A [survey](https://arxiv.org/abs/2504.14891) frames the two core problems as retrieving the ground truth and generating a response that aligns with the gold answer. Run both against a fixed test set and change one variable at a time.

### What are the main RAG evaluation metrics?

For retrieval: Recall@K, Precision@K, MRR and NDCG. For generation: faithfulness (does the answer reflect the retrieved documents), answer relevance and correctness against a reference. Definitions are in the [survey by Gan and colleagues](https://arxiv.org/abs/2504.14891).

### Can you evaluate RAG without human-labeled data?

Partly. Es and colleagues [propose](https://arxiv.org/abs/2309.15217) reference-free metrics that need no human-annotated ground truth. Saad-Falcon and colleagues [report](https://arxiv.org/abs/2311.09476) that LLM judges corrected with a small human-labeled set stay accurate with only a few hundred labels.

### How many test questions do you need?

Hugging Face's [cookbook](https://huggingface.co/learn/cookbook/en/rag_evaluation) recommends generating more than 200 synthetic question-answer pairs for a real knowledge base, because about half get filtered out. Google Cloud's [guide](https://cloud.google.com/blog/products/ai-machine-learning/optimizing-rag-retrieval) says the set should cover a broad subset of the underlying data and vary phrasing and question complexity.