How to test AI agents

To test an AI agent, split the work into layers. Test the tools and the code around the model with ordinary unit tests (the same layers apply to any LLM application) and no LLM. Script or stub the model to check the control flow. Run a real model against mocked tools to check its decisions. Then run evals in CI and watch production for what the tests miss. A single layer cannot cover an agent, because the model is the one part that does not behave the same way twice.

Testing and evaluation overlap. Evaluation scores quality across many runs and is covered in how to evaluate AI agents and LLM evaluation. What follows is the test layers around it, and what to fake at each one.

Why ordinary tests are not enough

Camunda’s agent testing guide opens by stating that AI agent processes are non-deterministic. An agent may call tools in a different order or word its answer differently on each run. Anthropic makes the same point for evals: a task that passed on one run might fail on the next, so it runs multiple trials.

The practical rule is to push as much as possible into layers where the model is removed, and keep the model-dependent layers few and tolerant.

The test layers

Layer	What is real	What is faked	What it checks
1. Tool tests	Your tool code	No model at all	Each tool returns the right result and handles bad input
2. Scripted-model tests	Your agent loop, tools and output validation	The model, replaced by a stub or your own function	Control flow, tool arguments, retries and failure handling
3. Live model, mocked tools	The model and the agent loop	External tool calls, answered with sample data	Whether the model picks sensible tools and recovers from errors
4. Evals	Model, tools and a clean environment	Often a simulated user	Task success and reliability over several trials
5. Production monitoring	Everything	Nothing	Drift and failures your test set never contained

Layers 1 and 2 are repeatable. Layers 3 to 5 are not, so they need tolerant assertions and repeated runs.

Layer 1: test tools without the model

Tools are normal functions or API calls. Test them as you would any code, with no LLM involved. Pydantic AI’s testing guide says unit tests check your code around the model, meaning tools, dependencies, output validation and failure handling, without making model requests, so results are repeatable.

Layer 2: script the model

To test the agent loop, replace the model. Pydantic AI documents two stand-ins (source):

• TestModel by default calls every tool, then returns text or structured output generated from the tool and output schemas, so the values are placeholders.
• FunctionModel replaces the LLM with your own function that receives the message history. Use it when you need to control the tool arguments the “model” sends.

Two more pieces make this practical. Agent.override(model=TestModel()) swaps the model without touching the code that calls the agent, and models.ALLOW_MODEL_REQUESTS = False guards against an accidental real LLM call. capture_run_messages captures the messages of a run so a test can assert the sequence of tool calls, tool returns and responses.

The same pattern exists outside Python: stub the model, then assert on the calls it made. The guide is explicit about the limit. These tests do not measure answer quality, which is what evals are for.

Layer 3: a live model with mocked tools

Here the model is real and the world around it is not. Camunda’s guide takes this approach: it keeps the AI agent and LLM interaction real while mocking external tool executions. Outbound connectors are disabled and the test completes tool tasks with sample data. This removes network flakiness and side effects while still exercising the model’s decisions.

Because the agent can call tools in any order, a fixed script breaks. Camunda registers conditional behaviors that react to whichever tasks become active instead of expecting a fixed sequence. A behavior for a tool the agent never calls simply never fires.

Mocked tools also let you assert what was called. Anthropic notes that code-based graders can verify tool calls, such as which tools were used and with which parameters.

Asserting on free-text output

You cannot compare a live model’s wording to a fixed string. Camunda offers two assertion types (source):

Assertion	How it works	Trade-off
Judge	An LLM scores a variable against a natural-language expectation from 0.0 to 1.0. The default pass threshold is 0.5	Handles open-ended or multi-part criteria, but costs one extra LLM call each
Semantic similarity	Compares embeddings of the output and a reference string by cosine similarity, default threshold 0.5	Described as a deterministic, lower-cost alternative, but works on serialized strings, so not on documents or binary content

Judge-based checks bring their own calibration problem; see inter-annotator agreement for how to check a judge against people.

Layer 4: evals in CI

Evals add repeated trials and graders on top of the tests. Anthropic suggests two suites (source): capability evals, which start at a low pass rate, and regression evals, which should be near 100%. Automated evals can run on each agent change and model upgrade, as the first line of defense against quality problems. For gating a pull request on them, see LLM regression testing in CI.

Keep the environment clean. Anthropic warns that shared state between trials can cause correlated failures due to infrastructure flakiness rather than agent performance, so each trial should start from a clean environment. Grader choice, pass^k and building the task set are in how to evaluate AI agents.

Layer 5: production

A test set is only what you thought of. Anthropic lists production monitoring as the way to detect distribution drift and failures synthetic evals miss, and also names its limit: it lacks ground truth for grading. It also warns that automated evals can create false confidence if they do not match real usage patterns. Traces from live traffic are a source of new test cases; see agent observability and AI agent monitoring.

Security and safety tests

Adversarial checks are a separate set of tests, not a layer of the above. See AI red teaming and prompt injection.

Running agent tests in HoneyHive

HoneyHive’s experiments run a function, such as an agent, over a dataset with evaluators, and the docs list running experiments in CI/CD pipelines to catch issues before deployment. A custom run_id correlates results with CI pipelines, and the docs describe gating every pull request on evaluation metrics through GitHub Actions. See the CI regression detection docs for setup.

FAQ

How do you unit test an AI agent without calling the model?

Replace the model with a stand-in. In Pydantic AI, Agent.override(model=TestModel()) swaps the model, and setting models.ALLOW_MODEL_REQUESTS = False guards against accidental real LLM calls. These tests check your code around the model, not answer quality.

How do you test an agent when its output changes on every run?

Do not assert exact text. Camunda's guide says AI agent processes are non-deterministic, so its tests react to whichever tools the agent calls and check results with judge or semantic similarity assertions. Anthropic runs multiple trials because a task that passed once might fail on the next run.

What is the difference between testing and evaluating an agent?

Tests check that your code around the model behaves correctly and give repeatable pass or fail results. Evals measure how well the agent does its tasks, with graders and several trials. See how to evaluate AI agents.

Can agent tests run in CI?

Yes. Anthropic describes automated evals as running on each agent change and model upgrade as a first line of defense. Scripted-model tests need no model calls, so they are the cheapest to run on every commit.