What is agent observability?

Agent observability means recording each step an AI agent takes on a task (model calls, tool calls, plans, handoffs to other agents) as one connected trace, then measuring the run as a whole. It answers questions a log of single LLM calls cannot: whether the agent picked the right tool, whether it looped, why it stopped, and what the task cost.

The unit of analysis moves from one model call to the whole task, which can be dozens of calls the model chose as it went. For the single-request basics, see what is LLM observability.

What one agent trace looks like

An agent run is a tree: a root span for the task, and under it the plan, each model call, each tool call, and any other agent it hands work to. The agent and tool names below are placeholders.

invoke_agent support_agent
 |- plan support_agent
 |   '- chat <model>
 |- chat <model>              picks a tool
 |- execute_tool lookup_order args, result, error.type
 |- chat <model>
 |- execute_tool transfer_to_billing
 |   '- invoke_agent billing_agent
 |       |- chat <model>
 |       '- execute_tool refund
 '- chat <model>              final answer, finish: stop

The agent, plan and tool span names follow the OpenTelemetry GenAI agent spans; the chat <model> span name follows the GenAI client inference spans ({gen_ai.operation.name} {gen_ai.request.model}). The table below says which span records which behavior.

Agent behaviors and the spans that record them

Agent behavior	What the trace has to keep	Span in the OTel GenAI conventions
Multi-step loop	Every model call, in order, with its finish reason	chat {model} inference spans under invoke_agent {agent}
Planning	The call that produced the plan, apart from later steps	plan {agent}, a planning or task-decomposition phase (agent spans)
Tool calls	Tool name, arguments, result or error	execute_tool {tool}; arguments and result are opt-in attributes (agent spans)
Handoffs	Which agent took over	execute_tool with the “agent as a tool” refinement, “a tool call that transfers work or control to another agent”; target in gen_ai.transfer.target.name (agent spans)
MCP calls	Method and tool on both sides of the protocol	MCP spans, “compatible with GenAI [execute_tool spans]” (MCP spans)
Long sessions	All turns of one conversation	gen_ai.conversation.id on agent and inference spans (client inference spans)
Cost per task	Tokens summed over the run	gen_ai.usage.input_tokens and output_tokens per inference span (client inference spans)

Sources: client inference spans, agent spans, MCP spans. All three pages are marked Development, not stable. For the MCP side, see Model Context Protocol.

Agent failures and how they show in a trace

Hitting a step cap is not silent: LangGraph raises a recursion-limit error, the OpenAI Agents SDK raises MaxTurnsExceeded, and the Claude Agent SDK returns an error_max_turns result. The cap tells you that a run failed. The trace tells you why, because the child spans show the repeated calls.

Failure	What it looks like in the spans	Signal to compute per task
Loop	The same tool and arguments, repeated	Most calls with identical tool and arguments
Tool-argument error	execute_tool spans failing with a validation-type error.type	Tool errors grouped by tool and error.type
Runaway steps	Model calls piling up to a cap	Model calls near the cap; last finish reason
Premature stop	An answer before a needed tool ran	Fewer steps than usual, plus an answer check
Wrong tool	A plausible tool that succeeds but does not fit	An evaluator on tool choice
Cost blowup	One task spending several times the median	Tokens per task vs. median

How teams detect each failure

Published framework docs already expose most of these signals. The work is to record them on the trace and compare runs.

Failure	What the framework or standard gives you
Loop, runaway steps	LangGraph raises a recursion-limit error when a graph “reached the maximum number of steps before hitting a stop condition” (LangGraph). The OpenAI Agents SDK raises MaxTurnsExceeded past max_turns (OpenAI Agents SDK). The Claude Agent SDK returns a result with subtype error_max_turns and no final response (Claude Agent SDK). Count how often runs end this way.
Tool-argument or tool-name error	The OpenAI Agents SDK raises ModelBehaviorError when the model calls a tool the current agent does not have, and tool_not_found_behavior="return_error_to_model" keeps the run going (OpenAI Agents SDK). In OpenTelemetry, error.type is required on a failed operation (agent spans). A recovered error is normal. Watch for the same error repeating.
Wrong tool, premature stop	LangSmith defines trajectory evaluation as checking “the exact sequence of messages, including tool calls”, by match against a reference run (strict, unordered, subset or superset) or by an LLM judge (LangSmith). Find which cases to write with error analysis.
Cost blowup	The Claude Agent SDK result carries total_cost_usd, usage and num_turns, and a max_budget_usd cap stops the loop. Its usage field covers only the main loop, so use model_usage for the whole tree (Claude Agent SDK).
Handoff confusion	The gen_ai.transfer.target.name attribute names the agent a handoff targeted (agent spans).

What to look for in agent observability tools

Beyond the general criteria for LLM observability tools, check that a tool can:

1. Roll up model calls, tool errors and tokens per task, across sub-agents, and sort tasks by them.
2. Show steps as a sequence as well as a tree, so repeated calls stand out.
3. Filter a multi-agent trace by agent and show handoffs.
4. Store tool arguments and results, with redaction. Loops and argument errors are invisible without them.
5. Run evaluators on the whole trajectory (right tool, task finished) as well as on single outputs.
6. Group multi-turn sessions by conversation ID.

Agent observability in HoneyHive

HoneyHive stores each run as a session of nested model, tool and chain events. The Trajectory view plots each span as a bubble on a category-by-step grid, so repeated tool calls and spiking steps stand out. The Thread view groups a multi-agent session by agent and marks handoffs, and the Graph view has an Errors filter that shows only failed spans.

To set this up with a coding agent, install the honeyhive-instrument skill from the coding agents page and paste:

With honeyhive-instrument, trace every tool our agent calls with its
arguments and error type, and record the agent framework version.

FAQ

How is agent observability different from LLM observability?

LLM observability records and scores single requests. Agent observability keeps the whole task as one trace, because the agent chose its own sequence of model calls, tool calls and handoffs at run time. The per-request basics are in what is LLM observability.

Do I need OpenTelemetry for agent observability?

No, but it is the common format. The OpenTelemetry GenAI conventions define agent, plan, tool and model spans, and tools such as Datadog and HoneyHive read them. The conventions are still marked Development, so attribute names can change.

What should I alert on for an agent?

Alert on per-task numbers rather than per-call ones, such as runs that hit a turn or budget cap, tool errors grouped by tool, and cost per task against the median.