# What is distributed tracing?

> Distributed tracing follows one request across services by passing a trace ID with each call. For agents it also has to cross model, tool and agent boundaries.

Source: https://www.honeyhive.ai/learn/distributed-tracing (updated 2026-11-04)

Distributed tracing records the path of a single request as it moves through several services, so you can see which service did what and where time or errors came from. The OpenTelemetry primer [defines](https://opentelemetry.io/docs/concepts/observability-primer/) a distributed trace as one that "records the path taken by a single request (made by an application or end user) as it propagates through multiple services."

It matters most when you cannot reproduce a problem on your laptop. The same primer says tracing lets you debug behavior that is "difficult to reproduce locally." An [AI agent](/learn/what-are-ai-agents) that calls other agents and remote tool servers is a good example of that.

## How it works

A trace is made of spans. A [span](https://opentelemetry.io/docs/concepts/observability-primer/) is "a single unit of work or operation", with timestamps and attributes. Every span in a trace shares one trace ID, and each span records which span called it, its parent. Together the parent links form a tree.

The hard part is the hop between services. [Context propagation](https://opentelemetry.io/docs/concepts/context-propagation/) is "the mechanism that moves context between services and processes". The caller puts its trace ID and span ID in the outgoing request. The receiving service reads them and creates a new span that belongs to the same trace, with the caller's span as its parent.

By default OpenTelemetry uses the W3C `traceparent` header for this. The [W3C Recommendation](https://www.w3.org/TR/trace-context/) (published 23 November 2021) gives it four hex fields:

| Field | Length in hex characters | Holds |
|---|---|---|
| version | 2 | Format version |
| trace-id | 32 | The 16-byte trace ID |
| parent-id | 16 | The 8-byte ID of the calling span |
| trace-flags | 2 | Trace flags |

A second header, `tracestate`, carries "additional vendor-specific trace identification information across different distributed tracing systems". Because the format is a standard, services using different tracing tools can still join one trace. For the toolkit around it, see [what is OpenTelemetry](/learn/opentelemetry).

## Sampling: which traces you keep

Keeping every trace gets expensive at volume, so most systems sample. The [OpenTelemetry sampling docs](https://opentelemetry.io/docs/concepts/sampling/) describe two ways.

| | Head sampling | Tail sampling |
|---|---|---|
| When it decides | "As early as possible" | After seeing all or most spans of the trace |
| Strength | Easy to configure and efficient | Can keep traces by error, latency or attribute |
| Weakness | Cannot use data from the whole trace, so it cannot guarantee that every error trace is kept | Harder to run: needs stateful systems that handle large data volumes |

The docs add that for larger systems it is "almost always necessary to use Tail Sampling", and that tail samplers "often end up as vendor-specific technology today". If the traces you most need are the failed or slow agent runs, a head-only sampler can drop exactly those.

## Where agents change the picture

A classic web request crosses services you wrote. An agent run also crosses model calls, tool calls and other agents, and some of those sit behind protocols rather than plain HTTP.

The OpenTelemetry [GenAI agent span conventions](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md) split agent invocation by where the agent runs. An `invoke_agent` span of kind `CLIENT` covers an agent outside the current process, such as the OpenAI Assistants API, AWS Bedrock Agents or an A2A endpoint. An `INTERNAL` span covers an agent in the same process, such as in LangChain, CrewAI, [AutoGen](/learn/autogen-framework) or Google ADK. Only the first kind needs context to cross a boundary.

For tool servers, the [MCP conventions](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/mcp.md) say the client should inject context into the request's `params._meta` property bag, and the server should extract it from there and use it as the remote parent. The keys are `traceparent`, `tracestate` and `baggage`. The conventions note that MCP and transport contexts are independent, and that the MCP client span is the parent of the server span regardless of transport. See [Model Context Protocol](/learn/model-context-protocol) and [A2A](/learn/a2a-protocol) for the protocols themselves, and [multi-agent systems](/learn/multi-agent-systems) for why agents end up spread across services.

Both documents are marked Development as of November 2026, so attribute names can change.

## Why traces break

Each hop has to do two things: the caller injects the context, and the receiver extracts it. If either side skips its part, the receiver starts a new trace and one request shows up as two unrelated ones (see [context propagation](https://opentelemetry.io/docs/concepts/context-propagation/)). HoneyHive's docs list HTTP, queues and RPC as paths to cover when you [trace distributed systems](https://docs.honeyhive.ai/v2/tutorials/distributed-tracing.md), so check each one your agents use.

## Distributed tracing in HoneyHive

HoneyHive's [distributed tracing guide](https://docs.honeyhive.ai/v2/tracing/distributed-tracing.md) offers two approaches. Passing the session ID puts events from each service in one session, but as siblings rather than parent and child, which suits simple setups and Lambda. Context propagation sends the W3C trace context over HTTP headers so remote spans become children of the calling span. It provides `inject_context_into_carrier(headers, tracer)` for the client and `with_distributed_trace_context(headers, tracer)` for the server.

## FAQ

### What is the difference between a trace and a span?

A span is one unit of work, such as a model call or an HTTP request. A trace is the set of spans for one request. The OpenTelemetry primer [defines](https://opentelemetry.io/docs/concepts/observability-primer/) a span as "a single unit of work or operation".

### What is a traceparent header?

The W3C header that carries the trace ID and the caller's span ID from one service to the next. Its format is `version-trace-id-parent-id-trace-flags`, per the [W3C Trace Context Recommendation](https://www.w3.org/TR/trace-context/).

### Why do traces break between services?

Usually the context was not passed on. Each service starts a new trace unless the caller injects the context and the receiver extracts it, as the [context propagation docs](https://opentelemetry.io/docs/concepts/context-propagation/) describe.

### Do AI agents need distributed tracing?

Yes, once an agent calls another agent, a remote tool server or a separate service. Within one process, ordinary tracing is enough. See [agent observability](/learn/agent-observability) for what to record inside a trace.