Training & Adaptationintermediate12 min · 8 steps
Synthetic Data Generation
With high-quality human text running out, models now train largely on model-generated data. How to evolve it, critique it, verify it, filter it, and keep its lineage.
datarequestresponsesignalerror
ProblemStep 1 of 8
The data wall
Disclosed pretraining budgets already reach 15T–45T tokens, and the supply of high-quality, legally clean human text isn't growing that fast. That's the data wall.
So post-training sets are now mostly model-generated: reasoning traces, rewrites, verified solutions. Synthetic shares of pretraining are rising too. Microsoft's Phi-4 report gives 40% of pretraining tokens to synthetic data. Most labs don't publish their mix.
| Human data | Synthetic data | |
|---|---|---|
| Volume | Finite | Unbounded |
| Quality | Variable, noisy | Controllable |
| Cost | High (labellers) | Cheap (GPU inference) |
| Bias | Mirrors the internet | Can be balanced |
What changed
- +Human text supply (strained)
- +Phi-4's synthetic share
Lesson 6 of 8 in Training & Adaptation