# How to monitor AI agents in production

> Monitor an AI agent per task, not per call. Watch how runs end, cost and steps per run, tool errors and sampled quality scores, and alert on changes against a baseline.

Source: https://www.honeyhive.ai/learn/ai-agent-monitoring (updated 2026-11-01)

To monitor an AI agent in production, measure each task as a whole and alert when the numbers move. Watch four things: how runs end (success, turn cap, budget cap or error), how many model and tool calls and how much money a run takes, tool errors, and a sampled quality score on real traffic. Alert on change against a baseline as well as on fixed limits.

Here the focus is the numbers you watch and alert on. [Agent observability](/learn/agent-observability) covers the traces you open when an alert fires, and [LLM observability](/learn/llm-observability) covers single requests. An [agent](/learn/what-are-ai-agents) is harder to monitor than a plain LLM call because the model decides how many steps to take, so cost, latency and failure all vary per run.

## What changes when the thing you monitor is an agent

A single LLM call has one latency, one token count and one output. An agent run is a loop of model calls and tool calls that ends when the model stops asking for tools or a limit stops it. The [Claude Agent SDK docs](https://code.claude.com/docs/en/agent-sdk/agent-loop) say a quick question may take two or three turns while a complex task "can chain dozens of tool calls across many turns". The same prompt can therefore cost very different amounts, so per-call averages hide the runs that matter.

## Start with how runs end

Agent frameworks already report why a run stopped. The Claude Agent SDK returns a result subtype for every run ([agent loop docs](https://code.claude.com/docs/en/agent-sdk/agent-loop)):

| Result subtype | What happened |
|---|---|
| `success` | The task finished normally. |
| `error_max_turns` | The run hit the `maxTurns` limit before finishing. |
| `error_max_budget_usd` | The run hit the `maxBudgetUsd` limit before finishing. |
| `error_during_execution` | An error interrupted the loop, for example a cancelled request. |
| `error_max_structured_output_retries` | No valid structured output within the retry limit. |

The same docs say every subtype carries `total_cost_usd`, `usage`, `num_turns` and `session_id`. The `stop_reason` field can also be `refusal`, which you can count separately from errors.

Count each ending as a share of all runs per hour or day. A fixed alert on error runs catches outages. A change in the share of capped runs catches quieter problems, such as a prompt edit that makes the agent loop. Both limits default to no limit in this SDK, and the docs say that "Setting a budget is a good default for production agents." If your framework has no result subtype, record the end state yourself on the root span.

## Signals worth a chart

The OpenTelemetry GenAI conventions define agent-level metrics, all marked Development ([GenAI metrics](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md)):

| Signal | OpenTelemetry metric | Definition |
|---|---|---|
| Run duration | `gen_ai.invoke_agent.duration` | End-to-end duration of a single in-process agent invocation |
| Model calls per run | `gen_ai.invoke_agent.inference_calls` | Number of model calls the agent makes in one invocation |
| Tool calls per run | `gen_ai.invoke_agent.tool_calls` | Number of tool calls the agent makes in one invocation |
| Tool latency | `gen_ai.execute_tool.duration` | Duration of a single tool execution |

Cost per run is not in that list. Take it from the framework result (the Claude SDK reports `total_cost_usd`) or sum token counts over the run's spans. One trap in that SDK: the `usage` field "covers only the main agent loop", so use `model_usage` to count subagents too ([agent loop docs](https://code.claude.com/docs/en/agent-sdk/agent-loop)).

Watch the distribution of each signal, not the mean. The tail (a high percentile of model calls, cost or duration per run) is where loops and runaway tasks show up.

## Cost caps need headroom

A budget cap stops the loop but not instantly. The Claude SDK compares spend with the cap after each response arrives, so spend "can therefore pass the cap by up to the cost of that one response, plus anything that subagents still running at that moment spend". The docs say to leave headroom when setting it, and that the cap covers subagents ([agent loop docs](https://code.claude.com/docs/en/agent-sdk/agent-loop)). Alert before the cap, on the share of runs that get close to it, so a cap is a backstop and not your only control.

## Watch tool calls and what the agent is allowed to do

Tool calls are where an agent changes things outside the model, so monitor them as events. The Claude SDK's hooks fire before a tool runs (`PreToolUse`), after it returns (`PostToolUse`, listed for auditing outputs) and when subagents start and stop. They run in your application process and a `PreToolUse` hook can block a call ([agent loop docs](https://code.claude.com/docs/en/agent-sdk/agent-loop)). That gives you one place to log every call and count blocked ones. For blocking checks as such, see [AI guardrails](/learn/ai-guardrails).

The OpenAI Agents SDK takes the tracing route: it is "enabled by default", wraps each run in a trace, and puts function tool calls and handoffs in their own spans ([tracing docs](https://openai.github.io/openai-agents-python/tracing/)). Those docs also say `trace_include_sensitive_data` defaults to `True`, and when it is `False` Responses model spans leave out request input and response output. Decide what is allowed into your monitoring data before you ship, since the default records inputs and outputs.

## Quality is not in the error rate

A run can end in `success` and still give a wrong answer. Quality needs a score on real traffic, which means running evaluators on sampled production runs. For the evaluator types and when to use them, see [LLM evaluation](/learn/llm-evaluation). The monitoring decisions are which runs to score and what to alert on:

- Score a sample, not every run, if volume or evaluator cost is high.
- Keep every failed run, since failures are the traces you will open first.
- Alert on a drop in the score against last week, not only on a fixed floor.

## What to alert on

These pairings follow from the signals above. The thresholds are yours to set from your own baseline.

| Alert | Type | Why it catches problems |
|---|---|---|
| Share of runs ending in `error_max_turns` or `error_max_budget_usd` | Change vs. last period | Loops and stuck tasks |
| High-percentile model calls per run | Change vs. last period | Runaway steps before a cap is hit |
| High-percentile cost per run | Fixed limit and change | Cost blowups |
| Tool errors per tool | Fixed limit | A broken dependency or bad arguments |
| Sampled quality score | Change vs. last period | Silent quality drift after a prompt or model change |
| Runs near the budget cap | Fixed limit | Cap is about to become the control |

## Monitoring in HoneyHive

HoneyHive's [alerts](https://docs.honeyhive.ai/v2/monitoring/alerts/alerts_overview) have two types: Aggregate, which monitors "when metrics cross absolute thresholds", and Drift, which detects "when performance degrades compared to previous periods". The docs give an example of alerting when this week's latency is 20% worse than last week's, and say you can alert on latency, error rates, evaluator scores, usage costs, or any metric or metadata field in your events and sessions. Checks run on hourly, daily, weekly or monthly time buckets.

[Online evaluations](https://docs.honeyhive.ai/v2/monitoring/onlineevals) run any server-side Python or LLM evaluator asynchronously on ingested traces, filtered by event type, name or property. The sampling percentage sets what fraction of matching events get scored, and the docs suggest 100% under 1K events a day, 25 to 50% for 1K to 10K, and 5 to 25% above that. For trace volume, the [sampling guide](https://docs.honeyhive.ai/v2/tracing/sampling) says "Failed requests are the most valuable traces. Always capture them regardless of sampling rate." Its examples make sampling decisions per request, and it does not mention keeping whole sessions together, so check how a sampled agent session looks before relying on it.

## FAQ

### What is the difference between AI agent monitoring and agent observability?

Monitoring watches a fixed set of numbers and alerts when one moves. Observability keeps the full trace so you can find out why. You need both, and the trace work is covered in [agent observability](/learn/agent-observability).

### What is the most important thing to monitor on an agent?

How runs end. Agent loops can stop on success, a turn cap, a budget cap or an error, and the Claude Agent SDK [reports](https://code.claude.com/docs/en/agent-sdk/agent-loop) each as a separate result subtype. A rising share of capped runs is the earliest sign of a loop or a stuck task.

### Should I sample traces or evaluate every run?

Keep every failed run and sample the rest. HoneyHive's [sampling docs](https://docs.honeyhive.ai/v2/tracing/sampling) say to always capture failed requests, and its [online evaluation docs](https://docs.honeyhive.ai/v2/monitoring/onlineevals) suggest lower sampling percentages as daily volume grows.

### Are the OpenTelemetry agent metrics stable?

No. The [GenAI metrics](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md) are marked Development as of November 2026, so names and units can change.