RAG chunking strategies: how to split documents for retrieval
Chunking splits documents into pieces a retriever can find. Compare the main strategies, the evidence on each and sensible starting settings.
Chunking is how a RAG pipeline splits documents into pieces before embedding them, and the split changes what the retriever can find. The published evidence says a simple recursive splitter with sensible sizes is a strong default. Fancier methods such as semantic chunking cost more and have not shown consistent gains.
Anthropic notes that chunks are “usually no more than a few hundred tokens” and that chunk size, boundary and overlap can all affect retrieval performance. Microsoft adds a second reason to chunk: models have input token limits (8,191 tokens for text-embedding-3-small), and a long page on many sub-topics can be a poor fit for a single vector even when it fits.
The main strategies
| Strategy | How it splits | Use it when | Watch for |
|---|---|---|---|
| Fixed-size | A set number of tokens or characters, often with overlap | You want a fast baseline | Cuts through sentences and ideas |
| Recursive | Tries paragraph, then line, then word separators until chunks fit | Most text; LangChain’s docs say to start here | Separators and size still need tuning |
| Sentence-aware | Keeps sentence boundaries, as LlamaIndex’s SentenceSplitter does | Prose where mid-sentence cuts hurt | Chunk sizes vary |
| Document structure | Splits Markdown by headers, HTML by tags, code by functions | The source has real structure | Needs a parser per format |
| Semantic | Breaks where embedding similarity between sentences drops | Topics shift often inside long documents | Extra embedding cost; no consistent gain over fixed-size |
| Hierarchical | Several chunk sizes, children point to parents | You want small chunks to match and larger ones to answer | Needs a retriever that can merge, such as LlamaIndex’s AutoMergingRetriever |
| Sentence window | One sentence per node, with its neighbors stored as metadata | Precise matching with context at answer time | The window is hidden from the embedding model, so it adds context only after retrieval |
| Contextual | An LLM writes a short context for each chunk before embedding | Chunks lose meaning without their document | An LLM call per chunk |
Microsoft’s docs also list a custom combination: for large documents, use variable-sized chunks but append the document title to chunks from the middle so they keep their context.
What the evidence says
Strategy and size matter, and simple often wins. Brandon Smith and Anton Troynikov at Chroma tested recursive, token, semantic and LLM-based chunkers on five corpora with 472 generated queries. They reported that the choice can change recall by up to 9%. Recursive chunking at 200 tokens with no overlap did consistently well, and it beat the token splitter on every metric at sizes of 400 tokens or less with no overlap. The token splitter at 800 tokens with 400 overlap had 87.9 recall but only 1.4 precision. The authors note small corpora, LLM-written questions and no runtime measurement, so treat the numbers as one team’s results.
Semantic chunking is not a sure upgrade. Renyi Qu, Ruixuan Tu and Forrest Bao compared semantic and fixed-size chunking on document retrieval, evidence retrieval and answer generation (October 2024). They concluded the cost of semantic chunking is not justified by consistent gains. Chroma’s semantic chunkers did lead on some metrics in its own tests, so the result depends on the data and the implementation.
Overlap is a trade. Chroma found lower overlap improves IoU, which penalizes redundant tokens, while overlap can help recall with smaller chunks. Overlap also adds chunks: in Microsoft’s sample, 1,000-character chunks gave 172 chunks with no overlap and 216 with 200 characters of overlap.
Adding context to each chunk helps. In Anthropic’s September 2024 tests, prepending 50 to 100 tokens of generated context before embedding cut the top-20 retrieval failure rate from 5.7% by 35%, by 49% with contextual BM25 and by 67% with reranking. With prompt caching, Anthropic estimated a one-time cost of $1.02 per million document tokens under its stated assumptions (800-token chunks, 8,000-token documents).
Late chunking is the cheaper way to keep context. Michael Günther and colleagues proposed running all tokens through a long-context embedding model first and splitting afterwards, so each chunk embedding sees its neighbors. It needs no extra training. Carlo Merola and Jaspinder Singh compared late chunking with contextual retrieval in 2025: contextual retrieval preserved semantic coherence better but needed more compute, while late chunking was more efficient but tended to lose some relevance and completeness.
Starting settings
Sources disagree, which is a sign to test rather than copy.
| Source | Suggested start |
|---|---|
| Microsoft Azure AI Search | 512 tokens, 25% overlap (128 tokens); its Text Split skill suggests 2,000 characters with 500 overlap |
| Microsoft, fixed-size table | 200 words or 600 characters with 10 to 15% overlap |
| Chroma evaluation | 200 tokens, recursive, no overlap |
| Pinecone | Test 128 or 256 and 512 or 1024 tokens on representative queries |
Pinecone lists four factors to weigh: the type of data, the embedding model’s limits and training domain, the length and complexity of queries, and what consumes the results (search, question answering, RAG or an agent). Microsoft adds that the size should suit every model you use, such as both an embedder and a summarizer. More chunks in the prompt also use more of the context window.
A way to choose
- Begin with a recursive splitter at a few sizes (for example 200, 400 and 800 tokens) and low overlap.
- If the documents have headings, tables or code, split on that structure first, then size-limit within sections.
- If chunks read as orphaned fragments, try adding the document title or generated context, or late chunking if your embedding model supports long inputs.
- Try semantic chunking only if you can afford the extra cost, and compare it to the fixed baseline.
- Score each setting on the same real queries. Retrieval metrics such as recall@k tell you whether the right chunk came back, and that is what chunking changes. The method is in LLM evaluation.
If a single retrieve-then-answer pass still misses, let the model decide what to search, as described in agentic RAG. Frameworks ship these splitters ready to use in LangChain and LlamaIndex.
Check chunking in your traces
A chunking change shows up in retrieval, so look at what was retrieved for each query. HoneyHive’s LlamaIndex integration captures retrieval and embedding steps as spans, and its evaluator templates include Context Relevance, which checks whether the retrieved context is relevant to the query. Run the same queries before and after a chunking change and compare. More on the practice in LLM observability.
FAQ
What is the best chunking strategy for RAG?
There is no single best one. Chroma's evaluation found that recursive splitting at 200 tokens with no overlap performed consistently well, and its authors say simple heuristics often work when parametrized appropriately. Start there, then test on your own queries.
What chunk size should I use?
Microsoft's Azure AI Search docs recommend starting at 512 tokens with 25% overlap, while Chroma's tests favored 200-token chunks with no overlap. Pinecone suggests trying smaller chunks (128 or 256 tokens) and larger ones (512 or 1024 tokens) against representative queries.
Is semantic chunking worth it?
Not reliably. A 2024 paper comparing it with fixed-size chunking concluded that its computational costs are not justified by consistent performance gains.
What is late chunking?
It embeds the whole long text first with a long-context embedding model, then splits it into chunk embeddings before mean pooling, so each chunk keeps context from its neighbors (paper).
How do I know if my chunking works?
Measure retrieval separately from generation on real queries. See RAG pipeline and LLM evaluation.