How to generate synthetic test data for LLM evals

Synthetic test data for LLM evals is a set of inputs, and often expected outputs, that an LLM or script generates instead of collecting from users. Teams use it when there is little production traffic yet, or to probe a failure that rarely shows up in real logs. Hamel Husain and Shreya Shankar say to “use synthetic data to start error analysis before you have enough production traffic”. Real data is still better once you have it, and Anthropic’s advice for agent evals is to start from real failures: “Converting user-reported failures into test cases ensures your suite reflects actual usage.”

The test set is the input to LLM evaluation, so what is wrong with it carries into every score.

Methods

Method	How it works	Source
Dimensions and tuples	Pick the ways user queries vary, combine values into tuples, then turn each tuple into a natural-language query	Husain and Shankar
From documents	Take documents from your corpus, extract key facts, then generate questions those facts answer	Husain and Shankar
Seeds and bootstrapping	Start from a few seed tasks, have a model generate new ones, and filter invalid or near-duplicate outputs	Self-Instruct
Evolving existing cases	Rewrite known cases step by step into more complex ones	WizardLM (Evol-Instruct)

Dimensions and tuples

Husain and Shankar warn against just asking an LLM for test queries. They say “a structured approach using dimensions produces far better synthetic data”. A dimension is one type of variation in user behavior, and a tuple takes one value from each dimension. They suggest writing about 20 tuples by hand first.

To scale, generate tuples first, then convert each into a query in a separate prompt, which they say avoids repetitive phrasing. Tuples can come from a cross product filtered by an LLM, which covers more when most combinations are valid, or straight from the LLM, which gives more realistic combinations but “tends toward generic outputs and misses rare scenarios”.

They also say to fix obvious problems first: if the prompt lacks an instruction, fix the prompt instead of generating queries to test for it.

From documents, for RAG

For retrieval tests, Husain and Shankar describe taking documents from your corpus and extracting key facts. You then generate questions that each fact answers. In their words, “this reverse process gives you query-document pairs for measuring retrieval performance without manual annotation”. The generated questions come from the same documents the retriever searches, so read a sample and check that they sound like questions a user would ask, not like the passage they came from.

Seeds, bootstrapping and evolution

Self-Instruct, by Wang and colleagues (first submitted in December 2022), generates instructions, inputs and outputs from a language model, bootstraps off its own generations, and filters invalid or similar ones. Applied to eval data, the same pattern is a handful of real or hand-written cases as seeds, a generator that produces more, and a filter that drops duplicates and malformed cases. WizardLM’s Evol-Instruct starts from seed instructions and rewrites them step by step into more complex instructions, which is one way to make an easy test set harder.

Edge cases

OpenAI’s guide says to ensure test data “includes typical cases, edge cases, and adversarial cases”. Adversarial cases overlap with red teaming. For agents, Anthropic’s advice is to start from real failures, as above.

Checking the data

Generated data has errors, so treat it as a draft.

• Filter before use: drop malformed, duplicate and near-duplicate cases, as the Self-Instruct pipeline does.
• Balance the set. Anthropic says to “test both the cases where a behavior should occur and where it shouldn’t”.
• Have people review it. OpenAI’s guide, after suggesting an LLM to generate test data, says to use human expert labellers.
• Keep tasks unambiguous. Anthropic’s test is that “two domain experts would independently reach the same pass/fail verdict”. Where experts disagree, measure it with inter-annotator agreement.
• Compare with real data as soon as you have it. Husain and Shankar say to “compare synthetic examples with real data as soon as real data becomes available”.
• Match production traffic. OpenAI lists “creating eval datasets that don’t faithfully reproduce production traffic patterns” as an anti-pattern.

Where synthetic data misleads

Husain and Shankar say synthetic data “cannot tell you how common a failure is in production”, so it cannot set priorities by frequency. Synthetic sets can also be easier: the STELLAR-E authors (April 2026) found that “real datasets remain slightly more challenging for LLMs especially for smaller models”. Husain and Shankar list cases where it is unreliable:

• complex domain-specific content
• low-resource languages or dialects
• cases where realism cannot be validated
• high-stakes domains
• underrepresented user groups, where synthetic data can reinforce biases in the generating LLM’s training data

How to do it in HoneyHive

HoneyHive’s dataset docs do not describe a synthetic generator, so generate the cases with any of the methods above and bring them in. For behavior tests generated from a written spec, the ASSERT integration runs Microsoft’s open-source ASSERT, which turns a behavior specification into generated test cases and LLM judging, with the runs traced in HoneyHive. A dataset can come from selecting interesting interactions or edge cases in logged production data, from an upload of JSON, JSONL or CSV in the UI or with the Python SDK, or from YAML applied with the CLI (datasets overview).

Then run it as an experiment: evaluate(dataset_id=...) loads the datapoints and aggregates results while your function and evaluators run locally. Curate production traces into the same dataset as they arrive, then compare runs side by side, datapoint by datapoint.

FAQ

What is synthetic test data for LLM evals?

It is test inputs, and sometimes expected outputs, generated by an LLM or a script instead of collected from users. OpenAI's guide lists synthetic eval data alongside domain-specific and purchased data as a source for evals.

When should you not use synthetic data?

Hamel Husain and Shreya Shankar list complex domain-specific content, low-resource languages, high-stakes domains, underrepresented user groups and any case where you cannot validate realism. Synthetic data also cannot show how common a failure is in production.

How many test cases do you need?

Anthropic says 20 to 50 simple tasks drawn from real failures is a good start. Hamel Husain and Shreya Shankar suggest writing about 20 dimension tuples by hand before scaling up with an LLM.

Can an LLM generate the test data and also grade it?

The guides cited here do not measure that, but they add a review or filter step. OpenAI's guide, right after suggesting an LLM to generate test data, says to use human expert labellers, and the Self-Instruct pipeline filters invalid or similar generations before using them.