Synthetic Data for LLMs: Training Smarter with Generated Data

The best-performing small models today are trained largely on synthetic data generated by larger models. Synthetic data has moved from a workaround for data scarcity to a core strategy for building capable, efficient models.

Why Synthetic Data?

Real-world data has fundamental limitations:

• Scarcity: High-quality instruction-following data is expensive to collect at scale.
• Imbalance: Real data over-represents common tasks and under-represents rare but important ones.
• Noise: Human-generated data contains errors, inconsistencies, and low-quality examples.
• Privacy: Real user data often can’t be used for training due to privacy constraints.

Synthetic data addresses all of these — at the cost of introducing its own failure modes.

Self-Instruct

Self-Instruct (Wang et al., 2022) bootstraps instruction-following data from a base LLM:

1. Start with a small seed set of human-written instruction-response pairs.
2. Prompt the LLM to generate new instructions similar to the seed set.
3. Filter out low-quality or duplicate instructions.
4. Prompt the LLM to generate responses for the new instructions.
5. Add the new pairs to the training set and repeat.

This process generated the Alpaca dataset (52K examples) from GPT-3, which was used to fine-tune LLaMA into a capable instruction-following model.

Distillation-Based Synthetic Data

Use a large “teacher” model to generate training data for a smaller “student” model. The student learns to mimic the teacher’s outputs on a diverse set of inputs.

Orca (Microsoft) took this further by generating not just answers but also the teacher’s reasoning traces — teaching the student model to reason step-by-step, not just produce correct answers.

The Phi Models: Textbook-Quality Data

Microsoft’s Phi series demonstrated that data quality matters more than quantity. Phi-1 (1.3B parameters) was trained on:

• Filtered web data selected for educational quality.
• Synthetically generated “textbook” content covering programming concepts.
• Synthetically generated exercises with solutions.

Despite its tiny size, Phi-1 outperformed much larger models on coding benchmarks. The key insight: training on high-quality, pedagogically structured data produces better reasoning than training on the same volume of noisy web data.

Nemotron and Reward Model Data

NVIDIA’s Nemotron-4 used synthetic data to train a reward model for RLHF. Rather than collecting expensive human preference annotations, they generated preference pairs synthetically using a strong LLM as a judge.

This approach scales preference data collection to millions of examples at a fraction of the cost of human annotation.

Evol-Instruct and WizardLM

Evol-Instruct evolves existing instructions into more complex versions using a set of mutation operations:

• Add constraints (“solve this without using loops”).
• Increase complexity (“now handle edge cases X, Y, Z”).
• Deepen reasoning requirements (“explain why this approach is better than alternatives”).

WizardLM, trained on Evol-Instruct data, significantly outperformed models trained on the original instruction set.

Failure Modes and Risks

Model Collapse

Training on model-generated data and then using that model to generate more data creates a feedback loop. Each generation amplifies biases and reduces diversity. Eventually the model “collapses” to a narrow distribution. Mixing synthetic data with real data mitigates this.

Hallucination Propagation

If the teacher model hallucinates, the student learns to hallucinate the same things. Factual accuracy in synthetic data requires careful verification.

Benchmark Contamination

Synthetic data generated by a model that has seen benchmark data may inadvertently contaminate the student’s training, inflating benchmark scores.

Best Practices

1. Verify factual claims in synthetic data, especially for knowledge-intensive domains.
2. Maintain diversity — use diverse prompts and generation parameters to avoid mode collapse.
3. Mix with real data — pure synthetic training is risky; blend with high-quality real examples.
4. Filter aggressively — use quality classifiers to remove low-quality synthetic examples.
5. Track provenance — know which data is synthetic and which is real for debugging.

Synthetic data is now a standard tool in the LLM training toolkit. Used carefully, it enables capabilities that would be impossible to achieve with real data alone.