# Prefix caching

> Prefix caching reuses the KV cache of a prompt prefix that earlier requests already processed. Any change early in the prompt breaks the match from that point on.

Source: https://www.honeyhive.ai/learn/prefix-caching (updated 2026-11-22)

Prefix caching keeps the [KV cache](/learn/kv-cache) of prompts a server has already processed, so a later request that begins with the same tokens skips that work. The match is exact and runs from the first token, so one changed token early in the prompt breaks the reuse for everything after it. Put stable content first, and check the hit rate instead of assuming it.

Hosted providers sell the same mechanism as [prompt caching](/learn/prompt-caching). This page covers how the engines implement it, what invalidates a prefix and how to measure hits.

## How the engines do it

| | vLLM | SGLang |
|---|---|---|
| Name | Automatic prefix caching | RadixAttention |
| Structure | Hash per KV block | Radix tree over token sequences |
| Eviction | LRU | LRU on leaf nodes by default |
| Default | [On](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py) for models that support it; `--no-enable-prefix-caching` turns it off | [On](https://docs.sglang.io/docs/advanced_features/server_arguments); `--disable-radix-cache` turns it off |

vLLM [hashes](https://docs.vllm.ai/en/latest/design/prefix_caching/) each KV block from the parent block's hash, the block's tokens and extra values such as LoRA IDs, multimodal input hashes and cache salts. Only full blocks are cached, so the tail of a prefix that fills part of a block is recomputed. It evicts the least recently used blocks first.

SGLang's RadixAttention, introduced in the [SGLang paper](https://arxiv.org/abs/2312.07104) by Zheng and colleagues, stores KV tensors in a radix tree. The LMSYS team [wrote](https://www.lmsys.org/blog/2024-01-17-sglang/) that the tree keeps the cache for both prompts and generation results, so a later turn that includes an earlier answer can hit too. It evicts leaf nodes by least recent use and uses a cache-aware scheduling policy to raise the hit rate. SGLang's [server arguments](https://docs.sglang.io/docs/advanced_features/server_arguments) list other eviction policies, with `lru` the default.

Two engine-level details matter for multi-tenant serving. vLLM [supports](https://docs.vllm.ai/en/latest/design/prefix_caching/) a per-request `cache_salt` that is mixed into the first block's hash, so only requests with the same salt share blocks, which blocks timing attacks that infer cached content from latency. And its non-cryptographic `xxhash` option is faster but, in vLLM's words, theoretically raises collision risk, which could leak data in multi-tenant setups.

## What breaks a cached prefix

The rule is the same everywhere: the prefix must match token for token. These are the common causes, with what each provider documents.

| Change | Effect |
|---|---|
| A different system prompt, or anything dynamic near its top such as a date or user name | The match stops at the first changed token, so the rest is recomputed |
| Tool definitions edited or reordered | OpenAI [lists](https://developers.openai.com/api/docs/guides/prompt-caching) names, descriptions, schemas and ordering as things that can change the prefix. Anthropic [says](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) changing tool definitions invalidates the entire cache |
| Reasoning effort or thinking settings | OpenAI [says](https://developers.openai.com/api/docs/guides/prompt-caching) changing `reasoning.effort` can rewrite hidden system instructions. Anthropic [says](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) thinking parameters always invalidate message blocks and the effort setting can |
| `tool_choice` or images added or removed | Anthropic [says](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) this invalidates message blocks only |
| Unstable JSON key order in tool calls | Anthropic [warns](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) that some languages, such as Swift and Go, randomize key order when converting to JSON, which breaks caches |
| Editing earlier messages instead of appending | OpenAI [advises](https://developers.openai.com/api/docs/guides/prompt-caching) preserving earlier messages and tool results so later turns reuse the shared prefix |
| Request lands on a machine without the entry | OpenAI [says](https://developers.openai.com/api/docs/guides/prompt-caching) a request reuses a prefix only if it reaches a machine holding a matching, unexpired entry |

If you need to limit which tools a model uses without touching the prefix, OpenAI [suggests](https://developers.openai.com/api/docs/guides/prompt-caching) keeping the tool list fixed and using `allowed_tools`, or `tool_choice` to turn tools off. See [reasoning effort](/learn/reasoning-effort) for the setting itself and [system prompt](/learn/system-prompt) for what usually sits at the top.

## How to check hit rates

| Where | What to read |
|---|---|
| vLLM | Counters `vllm:prefix_cache_queries` and `vllm:prefix_cache_hits`, listed in the [metrics design doc](https://docs.vllm.ai/en/latest/design/metrics/). That doc says the metric of interest is hits per query |
| SGLang | The `--enable-cache-report` [flag](https://docs.sglang.io/docs/advanced_features/server_arguments), off by default, returns cached token counts in `usage.prompt_tokens_details` |
| Anthropic API | `cache_read_input_tokens`, `cache_creation_input_tokens` and `input_tokens`. Their [sum](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) is the total input, and if both cache fields are 0 the prompt was not cached |
| OpenAI API | `cached_tokens` under `usage.input_tokens_details`. On earlier models it is [computed](https://developers.openai.com/api/docs/guides/prompt-caching) from the last matched breakpoint and rounded down to a multiple of 128 |

Log cached and total input tokens on every request, then plot the cached share over time next to your deploys. A sudden drop after a prompt, tool or settings change points straight at the cause. The same ratio belongs next to [time to first token](/learn/time-to-first-token) on a production dashboard, since a lost cache means the prefix is recomputed during prefill, which is the phase prefix caching shortens.

## Where it helps and where it does not

vLLM's [docs](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/) name two workloads that benefit: repeated questions over the same long document and multi-round conversation. They also state the limit: prefix caching reduces prefill time, not decode time, so it brings no gain when most time goes to long answers or when queries share no prefix.

Agents are a good fit because every step resends the same system prompt and tool list. They are also where silent breaks happen, since a tool reorder or a settings change mid-session invalidates the prefix. For cost levers beyond caching, see [LLM cost optimization](/learn/llm-cost-optimization), and for the server side in general, [vLLM](/learn/vllm) and [LLM inference](/learn/llm-inference).

## FAQ

### What is prefix caching?

A serving optimization that keeps the [KV cache](/learn/kv-cache) of processed prompts so a new request that starts with the same tokens skips recomputing them. vLLM's [docs](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/) describe it as letting a new query reuse the cache of an existing query with the same prefix.

### What is the difference between prefix caching and prompt caching?

Prefix caching is the engine mechanism. [Prompt caching](/learn/prompt-caching) is the name hosted providers give to the same idea, sold with a discount and, for some providers, a write fee and a lifetime.

### What breaks a cached prefix?

Any change to the tokens before the point where requests should match. OpenAI [says](https://developers.openai.com/api/docs/guides/prompt-caching) reuse needs the entire rendered prefix to match, and a changed tool list or reasoning effort setting can alter that prefix.

### How do I check my prefix cache hit rate?

In vLLM, divide the `vllm:prefix_cache_hits` counter by `vllm:prefix_cache_queries`, which its [metrics design doc](https://docs.vllm.ai/en/latest/design/metrics/) lists. On hosted APIs, read the cached token fields in the response usage.

### Does prefix caching change the output?

No. vLLM's [design doc](https://docs.vllm.ai/en/latest/design/prefix_caching/) says prefix caching won't change model outputs. It saves prefill work only.

### Does prefix caching speed up long answers?

Not much. vLLM's [docs](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/) say it reduces prefill time but not the time spent generating new tokens.
