# RAG vs fine-tuning: which to use

> Use RAG when the model lacks facts, fine-tuning when it lacks consistent behavior. Start with prompts, then add one, and combine them only when evals show you need to.

Source: https://www.honeyhive.ai/learn/rag-vs-fine-tuning (updated 2026-11-04)

Use RAG when the model is missing information: private documents, recent facts, anything outside its training data. Use fine-tuning when the model has what it needs but does not behave the way you want: wrong format, wrong tone, inconsistent reasoning. OpenAI frames it the same way. RAG is context optimization, fine-tuning is LLM optimization, and "RAG only solves our in-context learning issues" ([OpenAI](https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy)).

Start with prompts. OpenAI [calls](https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy) prompt engineering "typically the best place to start".

## What each one changes

RAG leaves the model alone. It retrieves relevant text at query time and adds it to the prompt. See [RAG pipeline](/learn/rag-pipeline) for the stages.

Fine-tuning changes the model. IBM [describes](https://www.ibm.com/think/topics/rag-vs-fine-tuning) it as retraining a pretrained model on labeled data, which updates its weights.

| | RAG | Fine-tuning |
|---|---|---|
| What changes | The prompt, per query | The model's weights |
| Fixes | Missing, outdated or proprietary knowledge ([OpenAI](https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy)) | Inconsistent format, tone or reasoning ([OpenAI](https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy)) |
| New documents | Can be used within minutes ([AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-vs-fine-tuning.html)) | Retraining can take hours to days ([AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-vs-fine-tuning.html)) |
| Source citations | Response can reference its source ([AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-vs-fine-tuning.html)) | Fine-tuned models do not provide a source reference ([AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-vs-fine-tuning.html)) |
| Main cost | Data architecture to build and maintain ([IBM](https://www.ibm.com/think/topics/rag-vs-fine-tuning)) | Compute, labeled examples, and possibly a data scientist ([IBM](https://www.ibm.com/think/topics/rag-vs-fine-tuning), [AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-vs-fine-tuning.html)) |
| Known weakness | Weak at summarizing whole documents ([AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-vs-fine-tuning.html)); retrieval noise can hurt results ([OpenAI](https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy)) | Struggles to learn new facts ([Ovadia et al.](https://arxiv.org/abs/2312.05934)) |

## What the research says about knowledge

Ovadia and colleagues (submitted December 2023) compared unsupervised fine-tuning with RAG for adding knowledge to LLMs. Fine-tuning gave some improvement, but RAG consistently performed better, for knowledge the model had seen in training and for entirely new knowledge. They also found that LLMs struggle to learn new facts through fine-tuning, and that training on many variations of the same fact can help ([arXiv](https://arxiv.org/abs/2312.05934)).

This is one study, and it tested unsupervised fine-tuning. It supports a narrow point: do not fine-tune just to load facts into a model.

## What fine-tuning is good for

OpenAI lists the signs that fine-tuning fits: inconsistent results with incorrect formatting, an incorrect tone or style, and reasoning that is not followed consistently. It calls this "a learned memory problem" ([OpenAI](https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy)).

It can also cut prompt size. OpenAI says fine-tuning can minimize the tokens spent on prompt engineering and can reach the same accuracy with fewer tokens or a smaller model. In its own Icelandic correction example, BLEU rose from 62 for zero-shot GPT-4 to 87 for fine-tuned GPT-4 ([OpenAI](https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy)). That is one vendor's task, not a general result. For what BLEU measures, see [BLEU score](/learn/bleu-score).

AWS [adds](https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-vs-fine-tuning.html) that fine-tuning on proprietary or regulatory data can help a model follow in-house or industry standards, and recommends it when the model must do additional tasks such as summarization.

## How to decide

Follow this order, drawn from OpenAI, Anthropic and AWS guidance:

1. **Prompt first.** "Prompt engineering is typically the best place to start." See [prompt engineering](/learn/what-is-prompt-engineering) and [few-shot prompting](/learn/few-shot-prompting). For the prompting side of that choice, see [prompt engineering vs fine-tuning](/learn/prompt-engineering-vs-fine-tuning).
2. **Build an evaluation set.** OpenAI says to diagnose failures before choosing, and to get as much accuracy as you can from basic methods before RAG or fine-tuning ([OpenAI](https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy)). See [LLM evaluation](/learn/llm-evaluation).
3. **Read the failures.** Wrong or missing facts point to RAG. Right facts in the wrong format, tone or reasoning style point to fine-tuning.
4. **If the data is small, skip both.** Anthropic wrote in September 2024 that a knowledge base under 200,000 tokens (about 500 pages) can go straight into the prompt ([Anthropic](https://www.anthropic.com/news/contextual-retrieval)). Check your [context window](/learn/context-window).
5. **Question answering over your documents.** Start with RAG ([AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-vs-fine-tuning.html)).

## Using both

The two are not exclusive. OpenAI [says](https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy) "it's likely you may end up with fine-tuning + RAG in your production solution". AWS [describes](https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-vs-fine-tuning.html) the setup: the RAG architecture does not change, but the generating model is also fine-tuned.

RAFT, from a UC Berkeley team ([AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-vs-fine-tuning.html)), submitted March 2024, is a research method for this pairing. It trains the model to ignore irrelevant distractor documents and to cite the passage that supports its answer. The authors report consistent gains over baselines on the PubMed, HotpotQA and Gorilla datasets ([arXiv](https://arxiv.org/abs/2403.10131)).

When one retrieval pass is not enough, the model can decide when to search again. See [agentic RAG](/learn/agentic-rag).

## Measure either choice

Whichever you pick, score it on the same evaluation set before and after. [OpenAI's](https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy) guidance for fine-tuning is that data quality matters more than quantity, and that you can start with 50 or more examples. For RAG, retrieval needs tuning because extra noise can hurt. Both are testable changes, so treat them like any other change to an LLM system and compare results rather than trusting the approach.

## FAQ

### Is RAG or fine-tuning better?

Neither wins in general. They fix different problems. OpenAI [describes](https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy) RAG as context optimization and fine-tuning as LLM optimization. In one study of knowledge injection, RAG consistently beat unsupervised fine-tuning ([Ovadia et al.](https://arxiv.org/abs/2312.05934)).

### Can you use RAG and fine-tuning together?

Yes. AWS [says](https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-vs-fine-tuning.html) the RAG architecture stays the same while the generating model is also fine-tuned. RAFT is a research method that trains a model to work with retrieved documents ([paper](https://arxiv.org/abs/2403.10131)).

### Does fine-tuning teach a model new facts?

Poorly. Ovadia and colleagues found that LLMs struggle to learn new facts through fine-tuning, though training on many variations of the same fact can help ([arXiv](https://arxiv.org/abs/2312.05934)).

### What should I try first?

Prompt engineering. OpenAI [says](https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy) it is typically the best place to start, and to build an evaluation set before adding RAG or fine-tuning.

### Which is better for data that changes often?

RAG. AWS [says](https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-vs-fine-tuning.html) it can incorporate the latest documents in minutes, while fine-tuning can take hours to days.