Prefix caching

Monitor in production
HoneyHive Engineering

Prefix caching reuses the KV cache of a prompt prefix that earlier requests already processed. Any change early in the prompt breaks the match from that point on.

Prefix caching keeps the KV cache of prompts a server has already processed, so a later request that begins with the same tokens skips that work. The match is exact and runs from the first token, so one changed token early in the prompt breaks the reuse for everything after it. Put stable content first, and check the hit rate instead of assuming it.

Hosted providers sell the same mechanism as prompt caching. This page covers how the engines implement it, what invalidates a prefix and how to measure hits.

How the engines do it

vLLM SGLang
Name Automatic prefix caching RadixAttention
Structure Hash per KV block Radix tree over token sequences
Eviction LRU LRU on leaf nodes by default
Default On for models that support it; --no-enable-prefix-caching turns it off On; --disable-radix-cache turns it off

vLLM hashes each KV block from the parent block’s hash, the block’s tokens and extra values such as LoRA IDs, multimodal input hashes and cache salts. Only full blocks are cached, so the tail of a prefix that fills part of a block is recomputed. It evicts the least recently used blocks first.

SGLang’s RadixAttention, introduced in the SGLang paper by Zheng and colleagues, stores KV tensors in a radix tree. The LMSYS team wrote that the tree keeps the cache for both prompts and generation results, so a later turn that includes an earlier answer can hit too. It evicts leaf nodes by least recent use and uses a cache-aware scheduling policy to raise the hit rate. SGLang’s server arguments list other eviction policies, with lru the default.

Two engine-level details matter for multi-tenant serving. vLLM supports a per-request cache_salt that is mixed into the first block’s hash, so only requests with the same salt share blocks, which blocks timing attacks that infer cached content from latency. And its non-cryptographic xxhash option is faster but, in vLLM’s words, theoretically raises collision risk, which could leak data in multi-tenant setups.

What breaks a cached prefix

The rule is the same everywhere: the prefix must match token for token. These are the common causes, with what each provider documents.

Change Effect
A different system prompt, or anything dynamic near its top such as a date or user name The match stops at the first changed token, so the rest is recomputed
Tool definitions edited or reordered OpenAI lists names, descriptions, schemas and ordering as things that can change the prefix. Anthropic says changing tool definitions invalidates the entire cache
Reasoning effort or thinking settings OpenAI says changing reasoning.effort can rewrite hidden system instructions. Anthropic says thinking parameters always invalidate message blocks and the effort setting can
tool_choice or images added or removed Anthropic says this invalidates message blocks only
Unstable JSON key order in tool calls Anthropic warns that some languages, such as Swift and Go, randomize key order when converting to JSON, which breaks caches
Editing earlier messages instead of appending OpenAI advises preserving earlier messages and tool results so later turns reuse the shared prefix
Request lands on a machine without the entry OpenAI says a request reuses a prefix only if it reaches a machine holding a matching, unexpired entry

If you need to limit which tools a model uses without touching the prefix, OpenAI suggests keeping the tool list fixed and using allowed_tools, or tool_choice to turn tools off. See reasoning effort for the setting itself and system prompt for what usually sits at the top.

How to check hit rates

Where What to read
vLLM Counters vllm:prefix_cache_queries and vllm:prefix_cache_hits, listed in the metrics design doc. That doc says the metric of interest is hits per query
SGLang The --enable-cache-report flag, off by default, returns cached token counts in usage.prompt_tokens_details
Anthropic API cache_read_input_tokens, cache_creation_input_tokens and input_tokens. Their sum is the total input, and if both cache fields are 0 the prompt was not cached
OpenAI API cached_tokens under usage.input_tokens_details. On earlier models it is computed from the last matched breakpoint and rounded down to a multiple of 128

Log cached and total input tokens on every request, then plot the cached share over time next to your deploys. A sudden drop after a prompt, tool or settings change points straight at the cause. The same ratio belongs next to time to first token on a production dashboard, since a lost cache means the prefix is recomputed during prefill, which is the phase prefix caching shortens.

Where it helps and where it does not

vLLM’s docs name two workloads that benefit: repeated questions over the same long document and multi-round conversation. They also state the limit: prefix caching reduces prefill time, not decode time, so it brings no gain when most time goes to long answers or when queries share no prefix.

Agents are a good fit because every step resends the same system prompt and tool list. They are also where silent breaks happen, since a tool reorder or a settings change mid-session invalidates the prefix. For cost levers beyond caching, see LLM cost optimization, and for the server side in general, vLLM and LLM inference.

FAQ

What is prefix caching?

A serving optimization that keeps the KV cache of processed prompts so a new request that starts with the same tokens skips recomputing them. vLLM's docs describe it as letting a new query reuse the cache of an existing query with the same prefix.

What is the difference between prefix caching and prompt caching?

Prefix caching is the engine mechanism. Prompt caching is the name hosted providers give to the same idea, sold with a discount and, for some providers, a write fee and a lifetime.

What breaks a cached prefix?

Any change to the tokens before the point where requests should match. OpenAI says reuse needs the entire rendered prefix to match, and a changed tool list or reasoning effort setting can alter that prefix.

How do I check my prefix cache hit rate?

In vLLM, divide the vllm:prefix_cache_hits counter by vllm:prefix_cache_queries, which its metrics design doc lists. On hosted APIs, read the cached token fields in the response usage.

Does prefix caching change the output?

No. vLLM's design doc says prefix caching won't change model outputs. It saves prefill work only.

Does prefix caching speed up long answers?

Not much. vLLM's docs say it reduces prefill time but not the time spent generating new tokens.

Share