What is OpenAI Agents SDK? How it works and how to evaluate it

The OpenAI Agents SDK is OpenAI’s framework for building agent apps with “very few abstractions.” It is available for Python and TypeScript (JS docs); the details below are for Python. It has three primitives: agents (LLMs with instructions and tools), handoffs and agents as tools for delegation, and guardrails for validating inputs and outputs. It is the production-ready successor to OpenAI’s earlier experiment, Swarm, and ships with built-in tracing (docs). Install it with pip install openai-agents; the latest release is 0.24.0 (October 15, 2026) and it needs Python 3.10 or newer (PyPI, read October 30, 2026).

How the run loop works

Runner.run, run_sync and run_streamed start an agent and loop until it finishes (running agents):

1. The runner calls the model for the current agent.
2. If the output is final (text of the expected type and no tool calls), the loop ends.
3. If the model requests a handoff, the current agent changes and the loop reruns.
4. If the model calls tools, the runner runs them, appends the results and reruns.

“If we exceed the max_turns passed, we raise a MaxTurnsExceeded exception,” and max_turns=None removes the limit. The docs also describe error_handlers for max_turns, model_refusal and invalid_final_output, which let you return a controlled final output instead of an exception.

Agents SDK or the Responses API

By default the SDK calls the Responses API for OpenAI models and wraps it in a runtime. The docs say to call the Responses API directly when you want to own the loop, tool dispatch and state, or the workflow is short-lived. Use the SDK when you want the runtime to manage turns, tool execution, guardrails, handoffs or sessions. You can mix both in one app (docs).

Handoffs and agents as tools

A handoff is exposed to the model as a tool, so a handoff to “Refund Agent” is the tool transfer_to_refund_agent. The new agent takes over the conversation and sees the whole history unless you set an input_filter. Nested handoff history is an opt-in beta, off by default (handoffs). With agent.as_tool(), a specialist runs as a tool and a central agent stays in control, getting the specialist’s output back instead of handing off the conversation (tools).

Where guardrails run

Guardrails are attached to agents and tools, and they do not all run at the same points (guardrails):

Guardrail	Runs	Watch for
Input	Only for the first agent in the chain	By default it runs in parallel with the agent, which “may have already consumed tokens and executed tools before being cancelled”; set run_in_parallel=False to block first
Output	Only for the agent that produces the final output, after it completes	Does not support run_in_parallel
Tool	Every guarded function-tool call, before and after execution	“Tool guardrails do not apply to the handoff call”

A tripped guardrail raises InputGuardrailTripwireTriggered or OutputGuardrailTripwireTriggered. For checks around every custom tool call in a workflow with handoffs, the docs say to use tool guardrails rather than rely on agent-level ones. For the wider topic, see what AI guardrails are.

Failure modes in the official docs

Failure	What the docs say
MaxTurnsExceeded	The run used up its max_turns of LLM-call turns; raise the limit, fix the loop, or add a max_turns error handler (running agents)
ModelBehaviorError	The model produced malformed JSON for a tool call or an output_type, misused tools, or a non-streaming Responses call ended failed or incomplete (running agents)
ModelTimeoutError, ToolTimeoutError	A model call exceeded ModelSettings.timeout, or a tool exceeded its timeout with timeout_behavior="raise_exception" (running agents)
Guardrail runs after the agent started	Parallel input guardrails do not stop token use or tool calls already under way (guardrails)
Handoff target sees too much history	The receiving agent sees the full conversation unless you set an input_filter (handoffs)
Traces hold sensitive data	trace_include_sensitive_data defaults to true, so generation and function spans store inputs and outputs; set it to false to omit them (tracing)

What the SDK traces by default

Tracing is on by default. Each Runner call is wrapped in a trace named “Agent workflow” unless you set a name, and the SDK records spans for tasks, model turns, agents, LLM generations, function tools, guardrails and handoffs. Use trace() to put several runs in one trace, and group_id to link traces from one conversation. The default exporter sends traces to the OpenAI Traces dashboard in the background, so in a long-running worker call flush_traces() when you need delivery at the end of a job (tracing).

Two limits matter. Tracing is unavailable for organizations under a Zero Data Retention policy. And add_trace_processor() keeps the OpenAI exporter on, while set_trace_processors() replaces it, so traces go only where you send them. For background on what to capture, see what agent observability is.

How to evaluate an Agents SDK app

OpenAI describes two stages (agent evals):

• Trace grading, which “is the fastest way to identify workflow-level issues.” A grader assigns “structured scores or labels to an agent’s trace” (trace grading). You open Logs > Traces in the OpenAI dashboard, create a grader and run it on selected traces. Typical questions are whether the agent picked the right tool and whether a handoff happened when it should have.
• Datasets and eval runs, once you know what good looks like, to move “from individual traces to repeatable datasets and eval runs” for comparing prompts and tracking changes over time.

HoneyHive experiments “run a function on a curated dataset, score each output with evaluators” and compare runs in the dashboard (experiments). For the general method, see LLM evaluation.

Tracing an Agents SDK app in HoneyHive

The OpenAI Agents SDK docs list HoneyHive among the tracing integrations for the SDK (tracing). HoneyHive traces it with the OpenAIAgentsInstrumentor from OpenInference, so existing agent code runs unchanged. It captures agent runs, LLM calls with token usage, tool calls, handoffs and agents used as tools. Pass tracer_provider to the instrumentor and call tracer.force_flush() before the process exits. The integration page lists openai-agents 0.14.6 as last tested (April 2026), older than the current SDK release (integration docs):

from honeyhive import HoneyHiveTracer
from openinference.instrumentation.openai_agents import OpenAIAgentsInstrumentor

tracer = HoneyHiveTracer.init()  # Reads HH_INGESTION_API_KEY from your environment
OpenAIAgentsInstrumentor().instrument(tracer_provider=tracer.provider)

Setup, single-agent, handoff and agents-as-tools examples, and troubleshooting are on the HoneyHive OpenAI Agents SDK integration page.

FAQ

How do I install the OpenAI Agents SDK?

Run pip install openai-agents, set OPENAI_API_KEY, and import from the agents module (docs).

Is tracing on by default in the OpenAI Agents SDK?

Yes. Disable it with OPENAI_AGENTS_DISABLE_TRACING=1, set_tracing_disabled(True) or RunConfig.tracing_disabled. The docs say tracing is unavailable for organizations under a Zero Data Retention policy (tracing).

Can I send Agents SDK traces somewhere other than OpenAI?

Yes. add_trace_processor() adds a destination alongside the OpenAI exporter, and set_trace_processors() replaces the defaults (tracing). HoneyHive uses an OpenInference instrumentor instead (integration docs).

What is the difference between a handoff and an agent used as a tool?

A handoff changes the current agent, and the receiving agent sees the conversation history unless you filter it. An agent used as a tool is called like any other tool and returns a result to the calling agent, which stays in control (handoffs, tools).