LLM monitoring: what to track on a production LLM app

LLM monitoring means watching a few numbers on a production LLM app and alerting when they move. It is one part of LLMOps. Google’s SRE book names four golden signals for any service: latency, traffic, errors and saturation. They carry over to an LLM app, but each one has an LLM-specific catch. Errors hide inside HTTP 200 responses, saturation is measured in tokens per minute, and the cost of a request depends on its length. Add sampled quality scores and a watch on model retirement dates, and you have the whole list.

The list below covers the app as a whole. For the recorded traces you open when an alert fires, see LLM observability. For loops and per-run limits, see AI agent monitoring. For why inputs and outputs shift over time, see data drift vs concept drift.

The four golden signals for an LLM app

The SRE book defines latency as “the time it takes to service a request”, traffic as demand on the system “measured in a high-level system-specific metric”, errors as “the rate of requests that fail”, and saturation as how full the service is. It also says monitoring should answer two questions, “what’s broken, and why?”. Monitoring answers the first. Traces answer the second.

Signal	What to measure for an LLM app	The catch
Latency	Time to first token, time per output token, total duration	One average hides streaming behavior and output length
Traffic	Requests, input tokens and output tokens per minute	Token volume, not request count, drives cost and limits
Errors	Failed requests by error type, including failures after a 200	A wrong answer is not an error status
Saturation	Remaining requests, input tokens and output tokens before a rate limit, and spend against a cap	Limits are per model class and several can bind at once

The next sections take these in turn, using what model providers document. Anthropic’s and OpenAI’s docs are quoted because their behavior is public. Check your own provider’s pages, since limits and error shapes differ.

Errors: count by type, not by status code

Status codes group very different failures. Anthropic’s error docs list a 429 rate_limit_error for three cases: a rate limit, the monthly spend cap of the account’s tier, and a spend limit on the Claude Code workspace. The first clears when you wait. The spend-cap 429 “has no retry-after header and keeps failing until access resumes”, so retries only add noise.

Status	Error type	What it usually means for you
400	invalid_request_error	A bad request, and also the response when usage reaches a spend limit you set yourself
429	rate_limit_error	A rate limit, or a spend cap that will not clear on retry
500	api_error	An internal error; the docs say to retry with exponential backoff
504	timeout_error	A request that timed out while processing
529	overloaded_error	The API is temporarily overloaded, which “can occur when the API experiences high traffic across all users”

The rate limit docs add that a spend-cap 429 carries the code enforced_spend_limit_reached in error.details.error_code, and that usage then pauses until 00:00 UTC on the first day of the next month. Alert on that code separately from ordinary rate limiting: the first is an outage for your users, the second is backpressure.

Two more points change how you compute an error rate:

• The Anthropic SDK retries transient failures twice by default. If you only count errors your code sees after retries, you miss how often the provider is failing. Count attempts as well as final outcomes.
• With streaming, the docs warn that “an error can occur after the API returns a 200 response”, and normal error handling does not apply. A success rate computed from status codes alone overstates reliability on streamed responses, so record how each stream ended.

Keep the request-id header (also request_id in error bodies) on every failed call. Anthropic’s error docs ask for it when you contact support.

Saturation: watch headroom, not just failures

A 429 is the last stage of saturation. Anthropic returns response headers such as anthropic-ratelimit-requests-remaining, anthropic-ratelimit-input-tokens-remaining and anthropic-ratelimit-output-tokens-remaining, each with a matching -limit and -reset. Recording the remaining fraction per model class lets you alert before requests fail.

Three details from the rate limit docs affect what you chart:

• Limits are measured in requests, input tokens and output tokens per minute, and exceeding any one returns a 429 naming it. Chart all three.
• The API uses a token bucket, so capacity refills continuously, and “short bursts of requests can exceed the limit”. A per-minute average can look safe while bursts get rejected.
• Sharp growth can trigger acceleration limits. The docs say to “increase your traffic gradually and maintain consistent usage patterns”, which is worth knowing before a launch or a batch backfill.

Workspace-level limits can be set below the organization’s. The anthropic-workspace-id response header shows which workspace a request counted against, which helps when one team’s traffic starves another’s.

Tokens and cost: count cached tokens

Cost follows tokens, so traffic should be charted in tokens. One accounting trap comes from prompt caching. Anthropic’s rate limit docs say input_tokens covers only tokens after the last cache breakpoint, and give the total as cache_read_input_tokens + cache_creation_input_tokens + input_tokens. A dashboard that plots input_tokens alone falls when caching improves, which looks like less traffic when it is really a better cache hit rate.

For most Claude models, only uncached input tokens count toward the input rate limit, and the docs recommend monitoring the cache hit rate in the Console. Treat cache hit rate as its own metric. A drop usually means a prompt change broke a stable prefix, and it shows up as both higher cost and less headroom.

Break spend down by model, by API key or workspace, and by feature if you can tag requests. Without a breakdown, a cost alert says that something grew but not what.

Latency: split it before you alert on it

OpenAI’s latency guide says “generating tokens is almost always the highest latency step”, that cutting 50% of output tokens “may cut ~50% of your latency”, and that cutting 50% of the prompt “may only result in a 1-5% latency improvement”. The practical result is that total duration mostly tracks output length, so a latency alert can fire because answers got longer, not because the provider got slower.

The OpenTelemetry GenAI metrics separate the parts. On the server side they define histograms for time to first token and time per output token. On the client side, inference calls get their own metrics: gen_ai.client.inference.duration, gen_ai.client.inference.time_to_first_chunk and gen_ai.client.inference.time_per_output_chunk; the generic gen_ai.client.operation.duration should not be reported for inference calls. All are marked Development as of November 2026, so names may change. Time to first token is what a user waits through on a streamed answer (see time to first token); time per output token is the provider’s generation speed. Track both with output length next to them, and read percentiles instead of the mean.

OpenAI’s guide calls streaming “the single most effective approach” to cutting wait time, which also means your latency metric should be defined per stream, not per HTTP request.

Quality: the signal status codes cannot give

An LLM app can return a fast, well-formed, wrong answer, and none of the signals above move. Quality needs a score on real traffic. The usual sources are evaluators run on sampled production traces, and user feedback such as ratings. For the evaluator types, see LLM evaluation.

Sampling matters because judging every answer costs money. HoneyHive’s sampling docs say “Failed requests are the most valuable traces. Always capture them regardless of sampling rate”, and sample ordinary traffic at a fixed percentage. Whatever your tooling, keep every failure and score a sample of the rest. Alert on a change in the score against the previous period as well as on a floor, since quality can slide without crossing a fixed line.

Safety checks that block a response belong with AI guardrails. Monitoring only counts how often they fire.

Model retirement is a monitored event

Providers retire models, and a retired model fails every request. Anthropic’s deprecation page says requests to retired models fail, that deprecated models “are likely to be less reliable than active models”, and that it gives “at least 60 days’ notice before model retirement for publicly released models”. OpenAI’s deprecations page says affected customers are notified by email and in the documentation, and lists a shutdown date and, for most models, a recommended replacement.

Notice goes to people, not to dashboards, and platforms differ. Anthropic notes that Amazon Bedrock and Google-managed Claude models set their own retirement schedules. So make it a monitored fact:

• Log the model name returned on each response, not only the one you requested.
• Keep a list of the models you call and their retirement dates, and alert when one is inside your migration window.
• Use the provider’s usage export to find stragglers. Anthropic’s Console exports a CSV of usage “broken down by API key and model”.

Anthropic and OpenAI both advise testing the replacement on your own application before the retirement date, so run your evaluation set against it early. A model swap can also change quality and cost without any error, so watch the quality and token signals right after one.

Alert rules that get acted on

The SRE book says “every page should be actionable” and that alert rules “should be simple to understand and represent a clear failure”. Fewer rules that each need a human response beat a wall of thresholds. The pairings below follow from the signals above. Set the numbers from your own baseline.

Alert	Type	Who acts
Spend-cap error code appears	Any occurrence	Whoever owns billing and limits
Remaining input or output token headroom is low	Fixed limit	Platform owner, to raise limits or shed load
Provider error share (429 without cap, 500, 529, 504) rises	Change vs. last period	On call; check provider status and fallbacks
Time to first token or time per output token rises	Change vs. last period	Whoever owns the model call; check output length first
Cache hit rate drops	Change vs. last period	Prompt owner
Cost per request or per feature rises	Change vs. last period	Feature owner
Sampled quality score drops	Change vs. last period	Prompt or retrieval owner
A model in use is inside its retirement window	Date-based	Whoever owns model migrations

Setting this up in HoneyHive

HoneyHive’s monitoring dashboard combines production traces, evaluations and user feedback, so cost, latency and quality sit in one view, and you can build custom queries and save charts to explore trends. Its alerts come in two types: Aggregate alerts fire when a metric crosses an absolute threshold, and Drift alerts fire when performance worsens compared with a previous period, for example latency 20% worse than last week. They can watch latency, error rates, evaluator scores, usage costs, request volumes, or any metric or metadata field in your events and sessions, evaluated over hourly, daily, weekly or monthly buckets.

Provider-side values such as the limit headers and the response model name only show up as chartable fields if your instrumentation records them as event metadata, so add them there. For the tracing side, see OpenTelemetry.

FAQ

What is LLM monitoring?

LLM monitoring is watching a fixed set of numbers on a production LLM app, such as latency, errors, token spend, limit headroom and sampled quality scores, and alerting when one moves. It tells you that something changed. Finding out why takes recorded traces, which LLM observability covers.

How do you monitor an LLM in production?

Start from the four golden signals (latency, traffic, errors, saturation), count errors by type instead of by status code, track input and output tokens including cached tokens, score a sample of real answers, and alert on a short list of simple rules that each need a human response.

Is monitoring an LLM different from monitoring an agent?

Yes. An LLM app makes a bounded number of calls per request, so per-request signals are enough. An agent loops, so you also watch how each run ends and how many steps it takes, which AI agent monitoring covers.

Do I need to monitor which model version my app calls?

Yes. Providers retire models on a published schedule, and requests to a retired model fail. Anthropic's deprecation policy gives at least 60 days' notice for publicly released models, and its Console can export usage by API key and model so you can see what still calls an old one.