Synthetic data generation with LLMs: methods for training and RAG data

Synthetic data generation is the use of a model to write the data another model learns from, retrieves with or is tested on. For language models the data is usually instructions, responses, reasoning steps, documents or search queries. The methods below come from the papers that introduced them, with the numbers each reported and the date it was submitted.

The focus here is training and retrieval data. For generating test cases to evaluate an agent, see synthetic test data for LLM evals. For the definition and the privacy angle, see what synthetic data is.

Methods at a glance

Method	Idea	Reported result	Source
Self-Instruct	A model writes instructions, inputs and outputs, filters invalid or near-duplicate ones, then finetunes	33% absolute gain over vanilla GPT3 on Super-NaturalInstructions (December 2022)	Wang et al.
Evol-Instruct	An LLM rewrites existing instructions step by step into more complex ones	Human evaluators judged its instructions better than human-created ones on the paper’s test bed (April 2023)	WizardLM
Persona-driven	Prompt the model from the viewpoint of many personas to get varied data	A collection of 1 billion personas curated from web data (June 2024)	Persona Hub
Magpie	Give an aligned model only the chat template up to the user turn, so it writes the user query itself	4 million instructions generated, 300K selected as high quality (June 2024)	Magpie paper
Distillation	A stronger model supplies labels or rationales for a smaller one	A finetuned 770M T5 beat a few-shot prompted 540B PaLM on one benchmark using 80% of the data (May 2023)	Distilling step-by-step
Rejection sampling	Sample many outputs, keep those a checker or reward model accepts	Combining rejection samples from several models took LLaMA-7B to 49.3% on GSM8K against 35.9% for plain SFT (August 2023)	Yuan et al.

Generating instructions

Self-Instruct bootstraps from the model itself. The authors say human-written instruction data is limited in quantity, diversity and creativity, so their pipeline generates instructions, inputs and outputs, filters invalid or similar ones, and finetunes the original model. They call it an “almost annotation-free method”, and its output on GPT3 was on par with InstructGPT-001 on Super-NaturalInstructions.

Evol-Instruct targets difficulty. The WizardLM authors note that humans may struggle to write high-complexity instructions, so an LLM rewrites a starting set into harder ones.

Persona-driven prompting targets diversity. The Persona Hub paper conditions generation on personas to tap different perspectives in the model, and applies it to math and logic problems, user prompts, knowledge-rich text and tool definitions.

Magpie needs no prompt list. Its authors observed that an aligned model such as Llama-3-Instruct will write a user message when given only the template up to the user slot. They generated 4 million instructions with responses and kept 300K.

Size is not the only lever. LIMA fine-tuned a 65B model on only 1,000 curated prompts and responses and reported strong results, which the authors read as evidence that almost all knowledge comes from pretraining. Selecting data is part of the job, not only generating it.

Generating answers and reasoning

For answers, a stronger model often writes the labels or the rationale. Distilling step-by-step uses LLM rationales as extra supervision when training a small model.

Where a correct answer can be checked, sampling plus a filter works well. STaR generates rationales, retries with the correct answer revealed when the first attempt is wrong, fine-tunes on the rationales that reached correct answers, and repeats. Rejection sampling fine-tuning, in the Yuan et al. paper, found that samples with more distinct reasoning paths improved math reasoning more, and that mixing rejection samples from several models helped further.

Quality filtering

Every method above generates more than it keeps. The filters the papers describe fall into three kinds.

• Heuristic and similarity filters. Self-Instruct drops invalid and similar generations before training.
• Execution and verification. For code, the Llama 3 paper runs static analysis and unit tests and says “only dialogs that pass all checks are included in the final dataset”. Math has the same property: the answer can be compared with a known one, as in STaR and RFT.
• Reward models and model judges. The same paper says rejection sampling takes typically 10 to 30 outputs per prompt and uses a reward model to select the best candidate. It also prompts a Llama 3 checkpoint to rate samples on a three-point scale and to judge generated code against the original.

Filtering the source data matters too. The phi-1 authors trained a 1.3B code model on 6B tokens of web data selected for “textbook quality” plus 1B tokens of textbooks and exercises generated with GPT-3.5. It scored 50.6% pass@1 on HumanEval and 55.5% on MBPP.

How much of a model’s data can be synthetic

The Nemotron-4 340B report says over 98% of the data in its alignment process was synthetic, and that the team open-sourced the generation pipeline. That figure covers alignment data only, not pretraining, which is the setting Gerstgrasser et al. used to study collapse.

Model collapse

In May 2023, Shumailov et al. found that “use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear”. They add that data about genuine human interactions becomes more valuable as generated content spreads online.

A follow-up changes the practical advice. Gerstgrasser et al. (April 2024) confirmed that replacing real data with each generation’s synthetic data tends toward collapse, then showed that accumulating successive generations of synthetic data alongside the original real data avoids it, across model sizes, architectures and hyperparameters. The working rule is to add synthetic data to real data and never use it to replace it.

Synthetic data for RAG

For retrieval-augmented generation the useful synthetic data is query and document pairs for training the retriever or reranker. InPars prompts a large model with a few examples to write queries for documents. Retrievers finetuned only on that unsupervised data beat BM25, and ones finetuned on both supervised and synthetic data transferred better zero-shot.

Promptagator does the same from no more than 8 examples per task. Its dual encoders beat ColBERT v2, trained on MS MARCO, by more than 1.2 nDCG on average across 11 retrieval sets, and rerankers trained on the same data added 5.0 nDCG.

Both papers are from 2022 and used older models, so treat the margins as a method result and not a current benchmark. They fit a RAG pipeline where the corpus is private or narrow and no labeled queries exist. How you split documents changes which queries make sense, so see chunking strategies. If you are deciding between tuning a retriever and tuning the generator, RAG vs fine-tuning covers the trade-off.

Checking the result

Generated training data needs its own evaluation. Hold out real examples that no generated data touched, compare against a model trained without the synthetic portion, and read a sample by hand. The LLM evaluation page covers how to score outputs, and few-shot prompting covers writing the seed examples most of these methods start from. Check the license of the model that generates the data before training another model on its output.

Tracking synthetic data in HoneyHive

HoneyHive’s datasets documentation lists ways to create a dataset, including uploads of JSON, JSONL or CSV files. It does not describe a built-in generator, so generate data elsewhere and upload it as its own dataset, separate from cases taken from production traces. Then run experiments on both to see whether the synthetic portion moves the score on real traffic.

FAQ

What is synthetic data generation?

Using an algorithm, now usually a language model, to produce data instead of collecting it. For LLMs that means instructions, answers, reasoning steps, documents or search queries. See what synthetic data is for the definition.

What are the main methods for generating synthetic training data?

Self-Instruct, Evol-Instruct, persona-driven prompting, Magpie-style extraction from an aligned model, distillation from a stronger model, and rejection sampling against a verifier or reward model. The table above compares them with the paper that introduced each.

Does training on synthetic data cause model collapse?

It can when synthetic data replaces real data. Shumailov et al. found that training on model-generated content makes the tails of the original distribution disappear. Gerstgrasser et al. found that accumulating synthetic data alongside the original real data avoids collapse in their experiments.

Can you use synthetic data for RAG?

Yes, mainly to create query and passage pairs for tuning a retriever or reranker. InPars and Promptagator both prompt an LLM to write queries for documents and train on the result.