What is semantic caching?

Semantic caching stores an LLM’s answers and reuses one when a new query means the same thing as an earlier query, even if the wording differs. A normal cache needs identical text. As AWS describes it, a semantic cache converts each query to an embedding, looks for similar past queries, and returns the stored response when the match is above a similarity threshold, so the model is never called.

It is a cost and latency tool, and it is not free of risk: every cache hit is an answer the model did not write for this exact question.

How a request flows

AWS and Redis describe the same loop:

query -> embed -> search cache
  hit  (similar enough): return stored answer
  miss: call the LLM, store query vector + answer

On a miss, AWS says, the system calls the model and saves the query embedding with the response for next time. Redis says similarity is usually cosine similarity over vectors of 768 or 1,536 dimensions, and that thresholds are commonly 0.85 to 0.95.

AWS gives an example: “How do I install the VPN app on my laptop?” and “Steps to get VPN working on my computer” are different strings. An exact-match cache calls the model for each. A semantic cache calls it once.

Semantic caching vs other caches

	Exact-match cache	Prompt (prefix) caching	Semantic cache
Matches on	Identical request text	Identical prompt prefix	Similar meaning
Where it runs	Your stack	Model provider	Your stack
Model still generates an answer	No	Yes	No
Risk of a wrong answer	None	None	Yes, set by the threshold

Provider prompt caching is a different thing. OpenAI says it reuses work when requests share a prefix and “does not change how the model generates output tokens”. Anthropic requires 100 percent identical prompt segments and says the response is identical to what you would get without caching. It saves input cost and time to first token (see time to first token); it does not skip the model. The two can be combined.

What the numbers say

Published figures come from vendors or small studies, so treat them as upper bounds for query mixes with many repeats.

• AWS reports benchmark reductions of up to 86 percent in inference cost and up to 88 percent in latency, with hits returning in milliseconds rather than seconds. The page does not give the benchmark method.
• Fu Bang’s GPTCache paper reports 2 to 10 times faster responses on cache hits with OpenAI’s GPT service.
• Regmi and Pun report cache hit rates of 61.6 to 68.8 percent and API call reductions of up to 68.8 percent across query categories.
• Redis estimates the vector search adds 5 to 20 ms, and that on a miss you pay that plus the normal model latency.

The last point is the cost model: a miss is slower than having no cache. Savings depend on how often users ask the same thing, which only your own traffic can tell you.

Where it fits

AWS lists RAG assistants and copilots, where many users ask the same questions of a shared knowledge base (see RAG pipeline), and agents that repeat small lookups. It fits poorly where answers depend on the user, on live data, or on a long private conversation.

Caching is also one of the features some LLM gateways offer. You can run it in a gateway or in your own application code.

Where it goes wrong

Threshold. Microsoft’s docs explain both failure modes. Set it too high and the cache fills with near-duplicate entries from repeated misses. Set it too low and it returns answers that do not match what the user meant. Redis advises starting at 0.90 to 0.95, tracking false positives, and logging answers users reject. It adds that when false positives pass 3 to 5 percent, tuning the threshold alone will not fix it. Those figures are vendor guidance.

Conversation context. Microsoft warns that a cache keyed on the latest prompt alone gets follow-ups wrong. Its example: “What is the second largest?” asked after a question about lakes and after one about stadiums. The cache matches the text and returns the lake answer to the stadium conversation. Key the cache on part of the chat history as well.

Stale answers. Entries need a lifetime. Microsoft lists TTLs, a maximum cache size and a maximum item count. Redis suggests 5 to 15 minutes for fast-changing data such as prices, 24 hours for stable FAQs, and flushing related entries when the source data changes.

Security. OWASP’s 2026 entry on vector and embedding weaknesses names semantic caches. It says an attacker who can craft content just above or below the threshold can poison an entry so that it serves attacker text to every semantically equivalent query. OWASP also advises enforcing tenant scoping inside the index query, not as a filter after retrieval, so one customer’s cached answer is never served to another. See OWASP LLM Top 10.

Choosing the embedding model

The embedding model decides what counts as “similar”. Redis and Virginia Tech researchers report that small embedding models fine-tuned for one epoch on domain data beat open-source and proprietary baselines on precision and recall. Redis sells caching, so read that as a direction to test, not a result for your domain.

Tools

GPTCache, described in Fu Bang’s paper, splits a semantic cache into an embedding generator, a similarity evaluator, a vector store, cache storage and eviction, according to its README. The same split applies to any build. The README warns the API may change at any time and says new LLM APIs and models are no longer being added, so check its state before adopting it. Managed options exist on cloud platforms, including Amazon ElastiCache and Azure Cosmos DB.

Measuring a cache in production

A cache changes quality as well as cost, so a hit needs to be visible in the same traces as a model call. Record whether each request was a hit, the similarity score, and the matched entry, and sample hits for review. OWASP’s mitigations for the same layer ask for logs of query, returned IDs and similarity scores. With HoneyHive you can attach these as metadata and metrics on a span with enrich_span(), then filter hits and misses and run online evaluators on both (see LLM evaluation). The general practice is covered in LLM observability and LLM temperature explains why the same query can already produce different model answers.

FAQ

What is semantic caching?

A cache that matches queries by meaning instead of exact text. It embeds each query, searches for similar past queries, and returns the stored answer if the match clears a similarity threshold, so the model is not called, as AWS explains.

How is semantic caching different from prompt caching?

Prompt caching is run by the model provider and reuses the processing of an identical prompt prefix, then still generates a fresh answer. Anthropic says the response is identical to what you would get without it. A semantic cache sits in your stack and skips the model call entirely.

What similarity threshold should I use?

There is no universal value. Microsoft's Cosmos DB docs say setting it may take trial and error. Redis suggests starting at 0.90 to 0.95 and watching false positives, which is vendor guidance, not a tested default.

Can a semantic cache return the wrong answer?

Yes. A threshold that is too low returns answers that do not match the question, and a cache keyed on the last message alone can return an answer from a different conversation. OWASP also lists cache poisoning as a risk.

Does GPTCache still work?

It is the best-known open-source implementation, described in a 2023 paper. Its README says the API may change at any time and that new LLM APIs and models are no longer being added.