# How to trace AI agents

> To trace an AI agent, wrap each task in one root span, record every model call and tool call under it, and carry the trace context across every boundary.

Source: https://www.honeyhive.ai/learn/agent-tracing (updated 2026-11-06)

To trace an AI agent, put each task inside one root span, record every model call and tool call as a child span, and make sure the trace context follows the work wherever it goes: another thread, another service, another agent. The result is one trace per task that you can read from the first prompt to the final answer.

This page is the how-to. For what the spans mean and which failures they reveal, see [agent observability](/learn/agent-observability). For the general mechanics of trace IDs and spans, see [distributed tracing](/learn/distributed-tracing); for what a single model-call span records, see [LLM tracing](/learn/llm-tracing); for how traces and spans differ, see [trace vs span](/learn/trace-vs-span).

## Decide what the trace has to answer

An [agent](/learn/what-are-ai-agents) picks its own sequence of steps, so a trace that only logs model calls cannot show why a run went wrong. Before instrumenting, list what you need to read back later:

- Which tool the agent chose, with what arguments, and what came back.
- Which model call produced each decision, and why the run stopped.
- Which other agent took over, and what it did.
- Tokens and cost for the whole task, not one call.

Each item maps to a span you create or that your instrumentation creates for you.

## Step 1: Pick how you will instrument

There are three routes, and you can mix them.

| Route | What you get | What you still do |
|---|---|---|
| Auto-instrumentation for your model provider or framework | Spans for model calls, and with framework instrumentation often tool calls and handoffs, without changing agent code. [HoneyHive's docs](https://docs.honeyhive.ai/v2/tracing/introduction.md) say tracing is automatic for major LLM providers and vector databases, using OpenTelemetry semantic conventions | Add spans for your own logic |
| Manual spans in your code | Spans around functions you choose | Name and attribute them consistently |
| An OpenTelemetry Collector in front of a backend | Any app that exports OTLP can send traces to the same place | Configure the exporter and processors |

If you choose OpenTelemetry, use its [GenAI agent span conventions](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md) for names so traces stay readable in any backend. They are marked Development, so pin your instrumentation version. The background is in [what is OpenTelemetry](/learn/opentelemetry).

## Step 2: Open one root span per task

The root span is the unit you will filter, score and count cost against, so put it around the whole task, not around a single model call. In the OpenTelemetry conventions, `invoke_agent` is the operation to use "when an LLM drives the control flow through tool calls", and `invoke_workflow` when "the application defines the control flow" (both on an internal span named `{operation} {agent name}`, per the [agent spans spec](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md)).

If the agent holds a conversation, set `gen_ai.conversation.id` on the spans when it is readily available, so turns of one conversation can be grouped. Values that stay constant for a whole session, such as tenant or app version, belong on the session. HoneyHive's `enrich_session()` [sets them once](https://docs.honeyhive.ai/v2/tracing/enrich-traces.md) for every trace in the session, while `enrich_span()` writes to the current span.

## Step 3: Record model calls and tool calls under it

Each model call is a child span with its token counts and finish reason. Each tool call is its own span, named `execute_tool {tool name}`, with the tool name required and `error.type` set when it fails, per the [execute tool spec](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/execute-tool.md).

Two details are easy to miss:

- **Handoffs.** When a tool call transfers work to another agent, the spec has an "agent as a tool" refinement with `gen_ai.transfer.target.name` and a `gen_ai.transfer.mode` of `pass_control` or `return_to_caller`. Instrumentation should not report additional spans for these tool calls, and the refinement "MUST NOT be used for transfers that do not happen through a tool call".
- **Planning.** The spec says a `plan` span "SHOULD NOT be reported when the instrumentation cannot distinguish planning from generic reasoning or normal inference." If you cannot tell the difference, skip the span instead of guessing.

For your own code, HoneyHive's [custom spans](https://docs.honeyhive.ai/v2/tracing/custom-spans.md) use a `@trace` decorator that captures inputs and outputs and nests calls into a hierarchy. The same decorator works on async functions. For loop iterations or conditional blocks, `enrich_span_context()` opens a span as a context manager.

## Step 4: Decide what content to keep

Content is where agent traces get risky. In the OpenTelemetry conventions the message and instruction attributes (`gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.system_instructions`, `gen_ai.tool.definitions`) are opt-in, and the spec warns they may contain sensitive information, and for the input and output messages that they are "likely to contain sensitive information including user/PII data". Tool arguments and results carry a similar warning.

The trade-off is direct. Without arguments and results you cannot see loops or argument errors. With them you are storing user data. Common ways to hold both: record them and redact before export, or record metadata such as size and key count in place of large payloads, which HoneyHive's [sampling guide](https://docs.honeyhive.ai/v2/tracing/sampling.md) recommends for big objects.

## Step 5: Carry the context across every boundary

A trace splits wherever the context is not passed on. Per the [OpenTelemetry docs](https://opentelemetry.io/docs/concepts/context-propagation/), the caller sends a trace ID and span ID, and the receiver creates its span in the same trace. The default format is the W3C `traceparent` header. Instrumentation libraries usually handle this automatically; where none does, you inject and extract the context yourself with the Propagators API. Check these places in an agent:

| Boundary | What to do |
|---|---|
| Worker threads | A plain `ThreadPoolExecutor.submit()` does not carry context. HoneyHive's [multi-threading guide](https://docs.honeyhive.ai/v2/tracing/multithreading.md) copies the context per submit with `contextvars.copy_context()` and runs the function through `ctx.run` |
| Another service or a remote tool server | Inject the context into the request headers and extract it on the other side. HoneyHive offers [two options](https://docs.honeyhive.ai/v2/tracing/distributed-tracing.md): pass the session ID (events land in one session as siblings) or use W3C propagation for a parent-child tree |
| Queues and background jobs | Inject the context into the message as the carrier and extract it in the consumer, as the [Propagators API](https://opentelemetry.io/docs/concepts/context-propagation/) describes |
| Serverless functions | Flush before the handler returns. HoneyHive's [distributed tracing guide](https://docs.honeyhive.ai/v2/tracing/distributed-tracing.md) tells you to call `tracer.flush()` |
| Public endpoints | Treat incoming trace headers as untrusted. The OpenTelemetry docs [note](https://opentelemetry.io/docs/concepts/context-propagation/) that forged headers can manipulate tracing data |

For agent tools served over the [Model Context Protocol](/learn/model-context-protocol), the OpenTelemetry GenAI conventions include MCP spans, covered in [agent observability](/learn/agent-observability) and in [how to trace MCP servers](/learn/mcp-observability).

## Step 6: Sample so you keep the runs that matter

Agent traces are large, because one task can hold many spans with long text. Sampling by percentage is the simplest option, and HoneyHive's [sampling guide](https://docs.honeyhive.ai/v2/tracing/sampling.md) lists three strategies to combine: always trace errors, always trace high-priority requests, and trace a fixed share of the rest.

The catch is when the decision is made. OpenTelemetry's [sampling concepts](https://opentelemetry.io/docs/concepts/sampling/) say head sampling decides as early as possible and so cannot guarantee that every trace containing an error is kept. Tail sampling decides once the spans are in, using errors, latency or attribute values, but OpenTelemetry calls it "difficult to implement" and "difficult to operate". Some high-volume systems use head sampling first and tail sampling later in the pipeline.

## Step 7: Check that one run produced one trace

Run the agent on a task that calls a tool, hands off to another agent and, if you have them, crosses a service boundary. Then open the trace and check:

1. There is one root span for the task, not several.
2. Every tool call appears as a child, with its arguments and error status.
3. Handoffs show the target agent.
4. Spans from threads and other services hang under the right parent.
5. Token counts add up to a plausible task cost.
6. A forced failure shows up with an error type.

If spans are missing or sit at the top level, go back to the boundary table in step 5. Once traces look right, [evaluate](/learn/ai-agent-evaluation) whole runs on them, and read [error analysis](/learn/error-analysis-llm) for how to turn failed traces into test cases.

## Tracing agents with HoneyHive

HoneyHive's [tracing docs](https://docs.honeyhive.ai/v2/tracing/introduction.md) cover capturing LLM requests, tool calls and agent handoffs in one hierarchical trace, automatic tracing for major LLM providers and vector databases, custom spans with the `trace()` decorator, and distributed tracing that puts multi-service calls in one session. Its [Thread view](https://docs.honeyhive.ai/v2/tracing/thread-view.md) shows multi-turn conversations and handoffs across sub-agents and tools. The [multi-agent tutorial](https://docs.honeyhive.ai/v2/tutorials/multi-agent-cookbook.md) builds a support bot with tracing and evaluation end to end, and the [Collector integration](https://docs.honeyhive.ai/v2/integrations/opentelemetry-collector.md) accepts OTLP from any instrumented app.

## FAQ

### What is the difference between agent tracing and agent observability?

Tracing is the recording step: spans for each model call, tool call and handoff, linked into one trace per task. [Agent observability](/learn/agent-observability) is what you do with those traces, such as spotting loops, tool errors and cost per task.

### Should I record tool arguments and results in a trace?

Usually yes, because loops and bad arguments are invisible without them. The OpenTelemetry [execute tool spec](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/execute-tool.md) makes `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result` opt-in and warns they may hold sensitive data, so redact before you store them.

### Why is my agent trace split into several traces?

The trace context was not passed on. A new thread, queue message, HTTP call or remote tool server starts a new trace unless the caller injects the context and the receiver extracts it, as the [OpenTelemetry docs](https://opentelemetry.io/docs/concepts/context-propagation/) describe.

### How do I sample agent traces?

Keep every failed run and a fixed share of the rest. OpenTelemetry notes that head sampling [cannot guarantee](https://opentelemetry.io/docs/concepts/sampling/) that every trace with an error is kept, so decide after the run ends when you can.