What is distributed tracing?
Distributed tracing follows one request across services by passing a trace ID with each call. For agents it also has to cross model, tool and agent boundaries.
Distributed tracing records the path of a single request as it moves through several services, so you can see which service did what and where time or errors came from. The OpenTelemetry primer defines a distributed trace as one that “records the path taken by a single request (made by an application or end user) as it propagates through multiple services.”
It matters most when you cannot reproduce a problem on your laptop. The same primer says tracing lets you debug behavior that is “difficult to reproduce locally.” An AI agent that calls other agents and remote tool servers is a good example of that.
How it works
A trace is made of spans. A span is “a single unit of work or operation”, with timestamps and attributes. Every span in a trace shares one trace ID, and each span records which span called it, its parent. Together the parent links form a tree.
The hard part is the hop between services. Context propagation is “the mechanism that moves context between services and processes”. The caller puts its trace ID and span ID in the outgoing request. The receiving service reads them and creates a new span that belongs to the same trace, with the caller’s span as its parent.
By default OpenTelemetry uses the W3C traceparent header for this. The W3C Recommendation (published 23 November 2021) gives it four hex fields:
| Field | Length in hex characters | Holds |
|---|---|---|
| version | 2 | Format version |
| trace-id | 32 | The 16-byte trace ID |
| parent-id | 16 | The 8-byte ID of the calling span |
| trace-flags | 2 | Trace flags |
A second header, tracestate, carries “additional vendor-specific trace identification information across different distributed tracing systems”. Because the format is a standard, services using different tracing tools can still join one trace. For the toolkit around it, see what is OpenTelemetry.
Sampling: which traces you keep
Keeping every trace gets expensive at volume, so most systems sample. The OpenTelemetry sampling docs describe two ways.
| Head sampling | Tail sampling | |
|---|---|---|
| When it decides | “As early as possible” | After seeing all or most spans of the trace |
| Strength | Easy to configure and efficient | Can keep traces by error, latency or attribute |
| Weakness | Cannot use data from the whole trace, so it cannot guarantee that every error trace is kept | Harder to run: needs stateful systems that handle large data volumes |
The docs add that for larger systems it is “almost always necessary to use Tail Sampling”, and that tail samplers “often end up as vendor-specific technology today”. If the traces you most need are the failed or slow agent runs, a head-only sampler can drop exactly those.
Where agents change the picture
A classic web request crosses services you wrote. An agent run also crosses model calls, tool calls and other agents, and some of those sit behind protocols rather than plain HTTP.
The OpenTelemetry GenAI agent span conventions split agent invocation by where the agent runs. An invoke_agent span of kind CLIENT covers an agent outside the current process, such as the OpenAI Assistants API, AWS Bedrock Agents or an A2A endpoint. An INTERNAL span covers an agent in the same process, such as in LangChain, CrewAI, AutoGen or Google ADK. Only the first kind needs context to cross a boundary.
For tool servers, the MCP conventions say the client should inject context into the request’s params._meta property bag, and the server should extract it from there and use it as the remote parent. The keys are traceparent, tracestate and baggage. The conventions note that MCP and transport contexts are independent, and that the MCP client span is the parent of the server span regardless of transport. See Model Context Protocol and A2A for the protocols themselves, and multi-agent systems for why agents end up spread across services.
Both documents are marked Development as of November 2026, so attribute names can change.
Why traces break
Each hop has to do two things: the caller injects the context, and the receiver extracts it. If either side skips its part, the receiver starts a new trace and one request shows up as two unrelated ones (see context propagation). HoneyHive’s docs list HTTP, queues and RPC as paths to cover when you trace distributed systems, so check each one your agents use.
Distributed tracing in HoneyHive
HoneyHive’s distributed tracing guide offers two approaches. Passing the session ID puts events from each service in one session, but as siblings rather than parent and child, which suits simple setups and Lambda. Context propagation sends the W3C trace context over HTTP headers so remote spans become children of the calling span. It provides inject_context_into_carrier(headers, tracer) for the client and with_distributed_trace_context(headers, tracer) for the server.
FAQ
What is the difference between a trace and a span?
A span is one unit of work, such as a model call or an HTTP request. A trace is the set of spans for one request. The OpenTelemetry primer defines a span as "a single unit of work or operation".
What is a traceparent header?
The W3C header that carries the trace ID and the caller's span ID from one service to the next. Its format is version-trace-id-parent-id-trace-flags, per the W3C Trace Context Recommendation.
Why do traces break between services?
Usually the context was not passed on. Each service starts a new trace unless the caller injects the context and the receiver extracts it, as the context propagation docs describe.
Do AI agents need distributed tracing?
Yes, once an agent calls another agent, a remote tool server or a separate service. Within one process, ordinary tracing is enough. See agent observability for what to record inside a trace.