What is Pydantic AI? How it works and how to evaluate it

Pydantic AI is an open-source Python framework for building LLM agents, made by the team behind the Pydantic validation library. You define an agent with a model, tools and an output type, and the framework validates tool arguments and final answers against your types, prompting the model to retry when validation fails (overview). It emits OpenTelemetry traces for each run (observability).

The parts

Part	What it does
Agent	The typed agent loop: model, instructions, tools, output type
Tools	Functions registered with @agent.tool; signatures and docstrings become the schema, and arguments are validated before your code runs
Output	output_type such as a Pydantic model; a failed validation prompts the model to try again
Dependencies	deps_type injects things like database connections into tools and instructions
Capabilities	Reusable bundles of tools, instructions, hooks and model settings; MCP and web search are in core
Pydantic Evals	Grades outputs and tool-call trajectories on datasets or live traffic
Pydantic Graph	Typed graph control flow for when one agent loop is not enough
Pydantic AI Harness	Capabilities for memory, guardrails, sub-agents, planning and persistence

Source for every row: the overview.

1.x to 2.x

Date	Change
2026-05-20	First 2.0 beta, forked from 1.100.0, with most of the breaking changes (one more came in beta 4)
2026-06-23	2.0.0 stable, with no new breaking changes since the betas

Both dates are from the changelog. The migration guide lists what broke:

• The default end_strategy changed from 'early' to 'graceful'.
• builtin_tools=, history_processors=, instrument= and prepare_tools= moved into capabilities=[...], and mcp_servers= became toolsets=[...].
• openai: now means the Responses API; openai-chat: selects Chat Completions.
• usage.request_tokens became input_tokens, and response_tokens became output_tokens.
• Instrumentation defaults to span format version 5, and run spans report gen_ai.aggregated_usage.* instead of gen_ai.usage.*.

Retry layers

The retries page documents seven layers that do not share a budget:

Layer	What it re-attempts	Default
Transport	The same HTTP request	None until you install a retrying transport
Provider SDK	The same request, by the SDK client	Provider-specific; OpenAI’s SDK defaulted to 2 retries as of 2026-10-30
Durable execution	The whole model request, by the workflow engine	Unbounded on Temporal
Model fallback	Not a retry: moves the request to another model	No same-model retries
Tool	One tool call, by asking the model to correct it	1 per tool
Output	The final answer, by asking the model to correct it	1
Model-request hooks	The model request, when a hook raises ModelRetry	Draws on the output budget

The docs warn that these layers multiply: one logical call can send up to N x F x M x K network requests. Budgets are set like this (from the same page):

agent = Agent('openai:gpt-6.1-sol', retries=3)
strict_output = Agent('openai:gpt-6.1-sol', retries={'tools': 5, 'output': 1})

Evaluation with pydantic-evals

Pydantic Evals runs datasets of cases through your task and scores them with evaluators: built-in checks, LLM judges, custom Python, span-based checks on OpenTelemetry traces, and agentic evaluators that grade the sequence and arguments of tool calls. It does not depend on pydantic-ai, so it can score any function.

Online evaluation attaches the same evaluators to production or staging calls. Each evaluator has its own sample rate, and when its concurrency limit is reached, new evaluation requests are dropped rather than queued. For the general method, see LLM evaluation.

Tracing options

Pydantic AI spans cover each agent run, model request and tool call, and follow the OpenTelemetry GenAI semantic conventions 1.37.0 (observability).

Route	How
Logfire	logfire.configure() and logfire.instrument_pydantic_ai()
Logfire SDK, another backend	Set the OTLP endpoint and send_to_logfire=False
Plain OpenTelemetry	Configure the OTel SDK, then call Agent.instrument_all()

Another Python SDK with built-in OpenTelemetry tracing is Strands Agents. For what to monitor in agents generally, see agent observability.

Tracing Pydantic AI in HoneyHive

HoneyHive ingests the native OpenTelemetry spans: initialize the tracer and call Agent.instrument_all() (integration page):

tracer = HoneyHiveTracer.init()  # Reads HH_INGESTION_API_KEY from your environment
Agent.instrument_all()

Agent runs, LLM calls with token usage, tool calls with arguments and results, and multi-turn history are captured. The integration page lists pydantic-ai 1.90.0 (May 2026) as the last tested version and requires 1.x. The span format changed in 2.x (above), so check that page for 2.x support before upgrading a traced app. A coding agent can do the setup with HoneyHive’s installable skills (AI coding agents): ask it to use honeyhive-instrument to add tracing to the Pydantic AI app. Full setup is on the HoneyHive Pydantic AI integration page.

FAQ

Is Pydantic AI free to use?

Yes. The package is MIT licensed and installs from PyPI. You still pay your model provider for tokens. Logfire is one tracing option, not a requirement: the instrumentation is OpenTelemetry, so other backends work.

What is the difference between Pydantic and Pydantic AI?

Pydantic is the data validation library. Pydantic AI is an agent framework from the same team that uses Pydantic models to describe tool arguments and agent outputs, so tool calls and final answers are validated the same way (overview).

Do tutorials written for Pydantic AI 1.x still work on 2.x?

Often not unchanged. Several arguments moved into capabilities=[...], OpenAIModel became OpenAIChatModel, and openai: now means the Responses API (migration guide). Upgrade to the latest 1.x first and clear its deprecation warnings.