# What is a RAG pipeline?

> A RAG pipeline fetches relevant documents at query time and gives them to an LLM.

Source: https://www.honeyhive.ai/learn/rag-pipeline (updated 2026-11-03)

A RAG pipeline is the chain of steps that lets a language model answer from documents it was not trained on. Ahead of time, the documents are split into chunks, turned into embeddings and stored in an index. For each question, the system retrieves the most relevant chunks and adds them to the prompt ([Anthropic](https://www.anthropic.com/news/contextual-retrieval)).

RAG stands for retrieval-augmented generation. The term comes from Lewis and colleagues (first submitted 22 May 2020), who combined "pre-trained parametric and non-parametric memory for language generation" ([paper](https://arxiv.org/abs/2005.11401)). The model's weights are the parametric memory; the searchable index is the non-parametric one.

## The stages

Sources group the steps differently. Google Cloud's reference architecture has two subsystems, data ingestion and serving: ingestion parses, chunks, embeds and indexes the data, and serving retrieves grounding data and sends the augmented prompt to an LLM. This table breaks those into four stages and adds two common extra steps.

| Stage | When | What happens | Source |
|---|---|---|---|
| Data preparation | Offline | Raw data is parsed, formatted and divided into chunks | [Google Cloud](https://docs.cloud.google.com/architecture/gen-ai-rag-vertex-ai-vector-search) |
| Indexing | Offline | Chunks become embeddings, stored in a vector database that supports semantic similarity search | [Anthropic](https://www.anthropic.com/news/contextual-retrieval) |
| Retrieval | Per query | The query is embedded and a vector-similarity search returns relevant grounding data | [Google Cloud](https://docs.cloud.google.com/architecture/gen-ai-rag-vertex-ai-vector-search) |
| Reranking (optional) | Per query | A reranking model scores each retrieved chunk for relevance and only the top ones reach the model | [Anthropic](https://www.anthropic.com/news/contextual-retrieval) |
| Generation | Per query | The augmented prompt, query plus retrieved data, is sent to an LLM | [Google Cloud](https://docs.cloud.google.com/architecture/gen-ai-rag-vertex-ai-vector-search) |
| Evaluation | Ongoing | Retrieval and answers are scored on real traffic | [Barnett et al.](https://arxiv.org/abs/2401.05856) |

The split matters for debugging. Offline stages run when your documents change. Per-query stages run on every request and set your latency.

## Decisions at each stage

**[Chunking](/learn/chunking-strategies).** Anthropic describes chunks as "usually no more than a few hundred tokens," and says chunk size, boundaries and overlap can all affect retrieval performance ([Anthropic](https://www.anthropic.com/news/contextual-retrieval)). A chunk cut away from its document can lose the context that makes it findable. Anthropic's contextual retrieval addresses this by adding context to each chunk before embedding.

**Retrieval.** Embedding search is one option. Anthropic's tests (published September 2024, across codebases, fiction, arXiv papers and science papers) combined contextual embeddings with BM25, a keyword method, and then added reranking. Against a baseline top-20 failure rate of 5.7%, contextual embeddings cut failed retrievals by 35%, adding BM25 by 49%, and adding reranking by 67% ([Anthropic](https://www.anthropic.com/news/contextual-retrieval)). These are one team's results on their own datasets, so test on yours.

**How many chunks to pass.** In Anthropic's tests, passing 20 chunks performed better than 5 or 10, though they recommend experimenting for your use case ([Anthropic](https://www.anthropic.com/news/contextual-retrieval)). More chunks also use more of the [context window](/learn/context-window).

**Generation.** The prompt combines the question with the retrieved text. How you word the instructions is [prompt engineering](/learn/what-is-prompt-engineering).

## Where pipelines fail

Barnett and colleagues (submitted 11 January 2024) identified seven failure points from three case studies in research, education and biomedical settings. Their conclusions: "validation of a RAG system is only feasible during operation," and "the robustness of a RAG system evolves rather than designed in at the start" ([paper](https://arxiv.org/abs/2401.05856)). A RAG pipeline is not finished when it first answers a question. It needs monitoring on real queries.

## Measure each stage

Score retrieval and generation separately, or you cannot tell which one caused a bad answer. Hamel Husain's evals FAQ treats retrieval as "a search problem" measured with "traditional information retrieval (IR) metrics", and generation as a check of "how well the LLM uses the retrieved context and whether it answers the question" ([evals FAQ](https://hamel.dev/blog/posts/evals-faq/)). Two checks:

- Retrieval: recall@k asks, "of all relevant documents, how many did you retrieve in the top k?" MRR asks "how high up was the first relevant document?" ([evals FAQ](https://hamel.dev/blog/posts/evals-faq/)). [Anthropic's](https://www.anthropic.com/news/contextual-retrieval) retrieval work uses 1 minus recall@20, the share of relevant documents missing from the top 20 chunks, as its metric.
- Generation: check "whether the answer is faithful to context", meaning the claims in the answer are supported by the retrieved chunks, alongside whether the context was relevant and whether the answer addresses the question ([evals FAQ](https://hamel.dev/blog/posts/evals-faq/)).

To score retrieval and generation separately, see [how to evaluate RAG systems](/learn/rag-evaluation). For the wider method, see [LLM evaluation](/learn/llm-evaluation).

## When you may not need one

If the whole knowledge base is under 200,000 tokens (about 500 pages), Anthropic says you can put it in the prompt and skip RAG, with prompt caching making that faster and cheaper ([Anthropic](https://www.anthropic.com/news/contextual-retrieval)).

When a single retrieve-then-answer pass is not enough, the model can decide when and what to search. See [agentic RAG](/learn/agentic-rag).

## Trace a RAG pipeline

To see which stage failed, record each one as a span. HoneyHive's [LlamaIndex integration](https://docs.honeyhive.ai/v2/integrations/llamaindex.md) uses the OpenInference instrumentor to capture query engine operations, LLM calls, retrieval and embeddings. In the docs' RAG example, the trace shows the query engine call, embedding calls, retrieval with document nodes, and the LLM synthesis step. Setup is `HoneyHiveTracer.init()` followed by `LlamaIndexInstrumentor().instrument(tracer_provider=tracer.provider)`, called before you create indexes, retrievers or query engines ([HoneyHive docs](https://docs.honeyhive.ai/v2/integrations/llamaindex.md)). More on the practice in [LLM observability](/learn/llm-observability).

To score the stages, HoneyHive has [evaluator templates](https://docs.honeyhive.ai/v2/evaluators/evaluator-templates.md) for Context Relevance (is the retrieved context relevant to the query), Answer Faithfulness (does the answer stay within the context) and Answer Relevance, which run server-side as LLM judges on your traces.

## FAQ

### What is a RAG pipeline?

It is the set of steps that lets an LLM answer from an external knowledge source. Documents are chunked, embedded and indexed ahead of time, then the most relevant chunks are retrieved for each query and added to the prompt ([Anthropic](https://www.anthropic.com/news/contextual-retrieval)).

### What are the stages of a RAG pipeline?

Data preparation (parse and chunk), indexing (embed and store), retrieval, and generation with an augmented prompt. [Google Cloud's](https://docs.cloud.google.com/architecture/gen-ai-rag-vertex-ai-vector-search) reference architecture covers these steps in two subsystems, data ingestion and serving. Reranking and evaluation are common additions.

### Who introduced RAG?

Patrick Lewis and colleagues, in a paper first submitted on 22 May 2020 and accepted at NeurIPS 2020 ([arXiv](https://arxiv.org/abs/2005.11401)).

### Do I always need RAG?

No. Anthropic [says](https://www.anthropic.com/news/contextual-retrieval) that if your knowledge base is smaller than 200,000 tokens (about 500 pages), you can include all of it in the prompt without RAG.

### How do you evaluate a RAG pipeline?

Score retrieval and generation separately. Retrieval is a search problem, so use recall@k, precision@k and MRR; for generation, check whether the answer is faithful to the retrieved context and whether it answers the question ([Hamel Husain's evals FAQ](https://hamel.dev/blog/posts/evals-faq/)).