# How to reduce LLM costs: levers, trade-offs and what to track

> Seven ways to cut LLM spend: prompt caching, batch APIs, model routing, context trimming, output limits, semantic caching and token tracking.

Source: https://www.honeyhive.ai/learn/llm-cost-optimization (updated 2026-11-06)

The way to cut LLM spend is to send fewer tokens, send them at a cheaper rate, or send fewer calls. Seven levers do that: prompt caching, batch APIs, smaller models and routing, context trimming, output limits, semantic caching and token tracking. Each one has a condition that decides whether it pays off.

Savings figures move with provider pricing, so the figures below are the providers' own multipliers as of November 2026, and dollar prices are left to the pricing pages.

## The levers at a glance

| Lever | What it changes | When it does not help |
|---|---|---|
| Prompt caching | Price of repeated prompt prefixes | Short prompts, prefixes that change every call, traffic too sparse to keep the cache warm |
| Batch API | Price per request, for work that can wait | Anything interactive |
| Smaller model or [routing](/learn/model-routing) | Price per token | Tasks where the small model fails and you pay twice |
| Context trimming | Input tokens per call | When trimming breaks a cached prefix or drops needed facts |
| Output limits | Output tokens per call | Tasks that need long answers |
| Semantic caching | Number of model calls | Queries that rarely repeat in meaning |
| Token tracking | Nothing directly | It is how you find where the other six apply |

## Prompt caching

Prompt caching lets the provider reuse the processed beginning of a prompt. The three big providers do it differently, and the differences decide the saving.

- **Anthropic.** You mark a breakpoint with `cache_control`, and the cache covers `tools`, `system` and `messages` in that order up to the breakpoint. The [docs](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) say a prompt below the model's minimum length is processed without caching and no error is returned, and the minimum depends on the model. As of November 2026 the docs list a cache read at 0.1 times the base input price for most models (lower on a few of the newest), a 5-minute write at 1.25 times and a 1-hour write at 2 times. A write costs more than a normal call, so a prefix you write and never read back loses money.
- **OpenAI.** Caching is [on by default](https://developers.openai.com/api/docs/guides/prompt-caching) for supported models, and the guide says to put stable instructions and shared reference material first and timestamps and user-specific text last, because reuse requires the whole prefix to match. It describes cached input as discounted up to 95% and warns that caching does not guarantee every request reuses stored data or runs faster.
- **Google.** [Implicit caching](https://aistudio.google.com/docs/caching) is on by default for Gemini 2.5 and newer models, and savings are passed on automatically when a request hits. Google advises putting large common content at the start and sending requests with similar prefixes close together in time.

The practical rules that follow from all three: order the prompt from most stable to least stable, keep anything that changes (a date, a user name, a retrieved passage) after the stable part, and do not reorder tools or edit the system prompt between calls. See [what a system prompt is](/learn/system-prompt) for what usually sits in that stable part.

## Batch APIs

If the work does not need an answer now, a batch endpoint is the simplest discount. OpenAI's [Batch guide](https://developers.openai.com/api/docs/guides/batch) lists a 50% discount compared with synchronous APIs, completion within 24 hours, separate rate limits, and up to 50,000 requests per file. Anthropic's [Message Batches docs](https://platform.claude.com/docs/en/build-with-claude/batch-processing) list a 50% reduction, with most batches finishing in under an hour (both as of November 2026). Anthropic's caching page adds that cache multipliers stack with the batch discount.

Good fits are nightly evaluation runs, bulk classification, backfills and synthetic data. Interactive chat and agent loops are not.

## Smaller models and routing

Per-token price differs a lot between a provider's largest and smallest models, so using the smallest model that passes your quality bar is often a large lever. OpenAI's [latency guide](https://developers.openai.com/api/docs/guides/latency-optimization) notes that smaller models are generally faster and cheaper and, with few-shot examples or fine-tuning, can hold quality.

Routing picks the model per query instead of per product. Two papers show the approach:

- The [FrugalGPT paper](https://arxiv.org/abs/2305.05176) (Chen, Zaharia and Zou, 2023) names three strategies: prompt adaptation, LLM approximation and LLM cascade. Its abstract reports matching GPT-4 with up to 98% lower cost, on the models available in 2023.
- The [RouteLLM paper](https://arxiv.org/abs/2406.18665) (Ong et al., 2024, revised in February 2025) trains routers on human preference data to choose between a stronger and a weaker model, and reports cost reduced by over 2 times in certain cases without hurting quality.

Both are results on their own benchmarks. Treat them as a method, and validate the split on your own traffic with [LLM evaluation](/learn/llm-evaluation) before routing real users. A cascade also pays for the cheap call when it falls through to the expensive one.

## Context trimming

Input tokens are paid on every call, and in an agent loop the history is re-sent each turn. Trimming means removing what the next call does not need: old tool results, stale reasoning, retrieved passages that were not used. A long window also degrades quality as it fills, which [context rot](/learn/context-rot) covers, so trimming can help twice. See [what a context window is](/learn/context-window) for the limits themselves.

Anthropic's [context editing](https://platform.claude.com/docs/en/build-with-claude/context-editing) does this server-side: it clears the oldest tool results once a token threshold is reached and keeps the full history on your side. The docs warn that clearing tool results invalidates cached prefixes and that you pay a cache write each time content is cleared, so they suggest a `clear_at_least` setting to make each clearing worth it. That is the cost of trimming and caching together: trim in large, infrequent steps, not a little on every turn.

Anthropic's [token counting endpoint](https://platform.claude.com/docs/en/build-with-claude/token-counting) returns the input size before you send a request, is free, and its description lists managing costs and routing decisions among its uses. It also notes that newer tokenizers can produce roughly 30 percent more tokens for the same text than earlier ones, so recount prompts against the model you plan to move to before estimating cost.

## Output limits

Output tokens are generated one at a time, and providers [price them](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) separately from input. OpenAI's latency guide frames shorter output mainly as a speed gain, with a rough heuristic that cutting 50% of output tokens may cut about 50% of latency. Its tips apply to cost as well: ask for concise answers, use few-shot examples that show short answers, and shorten field names in structured output. It also mentions ending generation early with `max_tokens` or `stop_tokens`. A hard cap truncates mid-answer, so set it above the longest answer you accept and prefer instructions for length.

## Semantic caching

Provider prompt caching reduces the price of a repeated prefix. [Semantic caching](/learn/semantic-caching) skips the call entirely by returning a stored answer when a new query means the same as an old one. It saves the most where users ask the same questions in different words, and it adds a risk that provider caching does not have: a wrong match returns a wrong answer. Tune the similarity threshold against labeled pairs, not by feel. An [LLM gateway](/learn/llm-gateway) is one place teams run it.

## Track token usage

You cannot choose among these levers without knowing where tokens go. The fields to log on every call:

- Input, output and cached tokens. Anthropic reports `cache_creation_input_tokens`, `cache_read_input_tokens` and `input_tokens`, and says total input is their sum. OpenAI reports `usage.input_tokens_details.cached_tokens`. Google reports `usage.total_cached_tokens`.
- The model, the feature and the user or tenant, so cost can be grouped.
- Cost per session or task, not only per call. An agent that makes 30 calls to finish one task is what you are billed for.

Two numbers tell you the most. A cache hit rate near zero on a long, stable prompt means something in the prefix changes between calls, or the prompt is under the model's minimum. A small group of sessions that account for most of the tokens is where trimming and routing pay off first. [LLM monitoring](/learn/llm-monitoring) covers what else to watch in production.

## Tracking cost in HoneyHive

HoneyHive takes token counts from the provider response on each traced LLM call and sums any cost you [attach](https://docs.honeyhive.ai/v2/tracing/aggregation-logic.md) to an event as `metadata.cost`. [Session aggregations](https://docs.honeyhive.ai/v2/tracing/aggregation-logic.md) roll cost, token and duration totals up to the session, the [monitoring dashboard](https://docs.honeyhive.ai/v2/monitoring/overview.md) shows cost, latency and token usage, [custom charts](https://docs.honeyhive.ai/v2/monitoring/charts.md) plot the trends by dimension, and [alerts](https://docs.honeyhive.ai/v2/monitoring/alerts/alerts.md) can fire on cost metrics. Start with the [tracing introduction](https://docs.honeyhive.ai/v2/tracing/introduction.md) to capture calls.

## FAQ

### What is the fastest way to reduce LLM costs?

Put the stable part of your prompt first so provider [prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) can reuse it, then check the cached-token fields in the response to confirm it is hitting. It needs no change to the model or the answers.

### Does the Batch API make LLM calls cheaper?

Yes, for work that can wait. OpenAI's [Batch guide](https://developers.openai.com/api/docs/guides/batch) lists a 50% discount against synchronous calls with a 24-hour window, and Anthropic's [batch docs](https://platform.claude.com/docs/en/build-with-claude/batch-processing) list 50% too, as of November 2026. It does not suit anything a user is waiting on.

### Can a cheaper model replace an expensive one without losing quality?

Sometimes, per query. The [RouteLLM paper](https://arxiv.org/abs/2406.18665) trains routers that choose between a stronger and a weaker model and reports cost cut by over 2 times in certain cases without hurting quality. You need your own evals to know where that holds for your traffic.

### What should I track to control LLM spend?

Input, cached and output tokens per call, grouped by model, feature and user, plus cost per session or task. Provider responses report cached tokens separately, so the cache hit rate is a number you can read directly.