What is a KV cache?

Monitor in production
HoneyHive Engineering

A KV cache stores the attention keys and values of earlier tokens so an LLM does not recompute them for every new token. It speeds up decoding and uses GPU memory.

A KV cache is a store of the key and value vectors that a transformer’s attention layers compute for tokens it has already processed. Hugging Face describes it as storing those pairs so they can be reused for later tokens instead of recomputed. It makes token-by-token generation faster, and it is a large part of why serving an LLM takes so much GPU memory.

Why it works

A model generates one token at a time, and each new token attends to all the earlier ones. With causal attention, a token’s representation never changes once it is processed, so the past keys and values can be kept. Hugging Face notes that at each step only the newest token’s query is needed, and its new key and value are appended to the cache.

The cache is kept per layer, as tensors shaped [batch_size, num_heads, seq_len, head_dim]. Hugging Face’s table compares the two modes: without a cache each step recomputes all previous keys and values, and with one it computes only the current token’s. Its docs say the cache should be used for inference only, since enabling it during training may cause errors.

Prefill and decode

A request has two phases, and the cache is built in the first. In the prefill phase the model processes the whole prompt and fills the cache. In the decode phase it generates tokens one by one, reading the cache each time and adding one entry. A longer prompt means more prefill work, which is why prompt length shows up in time to first token.

Why it uses so much memory

The cache grows with every token and holds an entry for every layer. The vLLM authors write that the KV cache memory of each request “is huge and grows and shrinks dynamically.” In their analysis of OPT-13B, one token needs 800 KB and one request can need up to 1.6 GB at a 2048-token maximum. On a 40 GB A100 serving a 13B model, close to 30% of memory held request states. These figures come from a 2023 paper and show the shape of the problem, not today’s sizes.

Hugging Face’s docs call the cache a potential bottleneck for long-context generation. The same pressure explains why a long context window costs more to serve than a short one.

Memory is also wasted by how the cache is stored. The vLLM paper reports that only 20.4% to 38.2% of KV cache memory held actual token states in the existing systems it measured, which the authors blame on fragmentation and redundant duplication. Its fix, PagedAttention, splits the cache into blocks managed like operating-system pages, and vLLM, built on it, reports 2 to 4 times higher throughput than FasterTransformer and Orca at similar latency.

Ways to shrink or manage it

Technique What it does Trade-off Source
Multi-query attention Shares one set of keys and values across all heads, so the cache is smaller and decoding reads less memory Minor quality loss reported Shazeer’s paper (2019)
Grouped-query attention Uses more than one but fewer key-value heads than query heads Quality close to multi-head attention at speed comparable to multi-query; an existing checkpoint can be uptrained with about 5% of pre-training compute Ainslie and colleagues’ paper (2023)
Paged allocation Stores the cache in blocks to cut fragmentation and allow sharing Needs a serving engine that supports it vLLM paper
Quantized cache Stores keys and values at lower precision Can hurt latency when context is short and memory is plentiful Hugging Face docs
Offloading Keeps one layer’s cache on the GPU and moves the rest to CPU Slightly lower throughput Hugging Face docs
Sliding-window attention Caps the cache on those layers at the window size Layers see only recent tokens Hugging Face docs

The first two change the model architecture, so you choose them when you pick a model. The rest are settings of the serving stack.

From KV cache to prompt caching

The cache normally disappears after one request. Providers can keep the cache for a shared prompt prefix and reuse it on later requests. OpenAI’s prompt caching guide says the prompt cache “stores key-value (KV) tensors, not the tokens themselves,” covering the full rendered context including tool definitions and conversation history, and that cached tensors may be held in GPU-local storage for at most 24 hours (as of November 2026).

Anthropic’s documentation describes the cached prefix as tools, then system, then messages, with a default lifetime of 5 minutes (as of November 2026) that is refreshed each time the content is used. It says caching has no effect on output generation: the response is identical to what you would get without it.

Because only a shared prefix can be reused, the order of your prompt matters. OpenAI’s latency guide advises putting stable instructions, examples and tool definitions first and, in multi-turn conversations, keeping the prefix unchanged and appending new messages. Manus’s Yichao Ji wrote in July 2025 that KV-cache hit rate is “the single most important metric for a production-stage AI agent.” See LLM cost optimization for how this affects spend and context engineering for keeping prompts stable. This is separate from semantic caching, which stores whole responses rather than attention states.

What to track

You do not see the KV cache from an API, but you can see its effects: input tokens, cached tokens, latency and cost per call. HoneyHive’s tracing concepts describe events that store model inputs, outputs and metadata such as token counts and cost, so you can compare calls with and without a stable prefix. More in LLM observability.

FAQ

What is a KV cache in an LLM?

It is a store of the key and value vectors that the attention layers compute for previously processed tokens. Hugging Face explains that the stored pairs are reused for later tokens instead of being recomputed.

Why does a KV cache speed up generation?

Without it, every step recomputes keys and values for all earlier tokens. With it, a step computes only the current token's. Hugging Face's docs describe attention cost per step as linear in sequence length with caching.

How much memory does a KV cache use?

It grows with every token and every request in the batch. The vLLM authors measured 800 KB per token for OPT-13B, up to 1.6 GB for one request at a 2048-token maximum.

Is a KV cache the same as prompt caching?

No, but they are related. A KV cache lives inside one generation. Prompt caching keeps the cache of a shared prefix between requests, and OpenAI's guide says its prompt cache stores KV tensors, not tokens.

Does the KV cache change the model's output?

Prompt caching does not. Anthropic's docs say the response is identical to what you would get without it. Quantizing the cache is a different matter, since it stores values at lower precision.

Share