What is LLM observability?

LLM observability is recording what an LLM application did on each request (the input, retrieved context, every model and tool call, the output, tokens and latency) as a trace, then scoring those traces for quality. It explains why a particular answer was wrong. Classic monitoring says whether the service responded; LLM observability says whether the response was any good and which step made it bad.

Why uptime and error metrics miss LLM failures

OpenTelemetry’s observability primer says observability lets you understand a system from the outside by asking questions about it. It also separates uptime from reliability: a system can be up 100% of the time and still be unreliable, because reliability asks whether the service does what users expect.

An LLM app can fail in exactly that way. In 2024, Air Canada’s website chatbot told a customer he could get a partial bereavement refund if he applied within 90 days, but the airline’s policy required requesting the discount before travel. Canada’s Civil Resolution Tribunal ordered the airline to pay CAD 812 in damages and tribunal fees and wrote that “it makes no difference whether the information comes from a static page or a chatbot.” There was no outage: the chatbot answered normally, and the answer was wrong.

The cause of such a failure sits in the content of the request, such as the retrieved document or the prompt. Error-rate and latency charts do not hold any of that, so a team needs the content recorded and judged.

What LLM observability adds to metrics, logs and traces

OpenTelemetry defines traces as the path of a request through your application, metrics as a measurement captured at runtime, and logs as a recording of an event. LLM apps keep all three and add a fourth kind of data, the quality score.

Signal	Classic web service	What an LLM app adds	Question it answers
Trace	Spans per service call, with timing and status	Spans for retrieval, model calls, tool calls and agent steps, with inputs and outputs	Which step produced the bad answer?
Metrics	Request rate, error rate, latency	Tokens, cost per request, evaluator pass rates over time	Is quality or spend drifting, and since when?
Logs	Errors and debug lines	Full prompts, retrieved chunks and model outputs, subject to privacy rules	What exactly did the model see?
Quality scores	Usually none	Code checks, LLM-as-judge results and human review attached to the trace they judged	Was this answer right for this request?
User signals	Clicks, sessions	Thumbs up or down, edits, retries, multi-turn conversations grouped as one session	Did the user get what they needed?

A quality score is the part with no real equivalent in classic observability. Cheap code checks can run on every trace; correctness and faithfulness judgments usually need a model or a person, so teams sample. See LLM evaluation for the judge side.

What a request looks like as a trace

session: support question
 |- chain: answer_question
     |- tool:  retrieval (docs returned, scores)
     |- model: generate  (prompt, output, tokens)
     |- scores: groundedness check, policy check

Each line is a span or event with its own input, output and timing. The scores attach to the step they judge, so a failed check points at retrieval or generation instead of at the whole app. Agents add more steps and loops: see agent observability.

What an LLM observability platform does

A platform stores these traces and does the work around them:

• ingests traces, often over OpenTelemetry;
• runs evaluators offline against test datasets and online against sampled production traffic;
• turns scores, tokens and latency into charts and alerts;
• lets people review and label individual traces;
• turns failures found in production into test cases for the next change.

For a side-by-side of platforms, see AI observability tools.

OpenTelemetry GenAI conventions: status as of 2026-10-30

OpenTelemetry’s GenAI semantic conventions define shared span names and attributes, so traces look alike across vendors. They now live in the semantic-conventions-genai repository, and the old page on opentelemetry.io carries a notice that it has moved.

Item	Status
Spec maturity	The GenAI spans spec is marked Development, not Stable
Operations defined	Model calls (chat, text_completion, generate_content, embeddings), agent and tool steps (create_agent, invoke_agent, invoke_workflow, plan, execute_tool), retrieval, memory operations, and fetch_response (attribute registry)
Signals covered	Spans, metrics and events for GenAI clients (repo README)

Because the spec is in Development, attribute names can change; check what your instrumentation library emits before building dashboards on them.

Tracing and scoring in HoneyHive

HoneyHive stores each request as a session of nested events of type model, tool or chain, each with inputs, outputs, timing, metrics and feedback, as the tracing concepts page describes. Online evaluations run Python or LLM evaluators on ingested traces, filtered by event type, name or property and sampled at a rate you set; scores appear on dashboards and on each trace.

To have a coding agent do the wiring, the coding-agent setup installs honeyhive-instrument for tracing and honeyhive-evaluate for evaluators. A prompt to paste: “Use honeyhive-instrument to trace our RAG pipeline, recording the IDs of retrieved documents. Then use honeyhive-evaluate to draft a Python evaluator for the retrieval event.”

FAQ

What is the difference between LLM monitoring and LLM observability?

Monitoring watches numbers chosen in advance, such as latency, error rate, token spend or an evaluator pass rate, and alerts when one crosses a threshold. Observability is what lets you work out why a number moved, or why one answer was wrong, from the recorded traces. Most teams run both.

Is LLM observability different from ML observability?

Yes, mostly in what gets recorded. ML observability for a classic model tracks feature and prediction distributions, drift and accuracy against labels that arrive later. LLM observability deals with free-text inputs and outputs, multi-step chains of prompts, retrieval and tool calls, and quality that has to be judged per answer, often by another model.