mlsanityData to 14 Oct 2026

Scaling Laws of Synthetic Data for Language Models

arXiv 2503.19551 ↗First posted 25 Mar 2025cs.CL cs.AIcited by 46

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Large language models (LLMs) achieve strong performance across diverse tasks, largely driven by high-quality web data used in pre-training. However, recent studies indicate this data source is rapidly depleting. Synthetic data emerges as a promising alternative, but it remains unclear whether synthetic datasets exhibit predictable scalability comparable to raw pre-training data. In this work, we systematically investigate the scaling laws of synthetic data by introducing SynthLLM, a scalable framework that transforms pre-training corpora into diverse, high-quality synthetic datasets. Our approach achieves this by automatically extracting and recombining high-level concepts across multiple documents using a graph algorithm. Key findings from our extensive mathematical experiments on SynthLLM include: (1) SynthLLM generates synthetic data that reliably adheres to the rectified scaling law across various model sizes; (2) Performance improvements plateau near 300B tokens; and (3) Larger models approach optimal performance with fewer training tokens. For instance, an 8B model peaks at 1T tokens, while a 3B model requires 4T. Moreover, comparisons with existing synthetic data generation and augmentation methods demonstrate that SynthLLM achieves superior performance and scalability. Our findings highlight synthetic data as a scalable and reliable alternative to organic pre-training corpora, offering a viable path toward continued improvement in model performance.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 49 of its 61 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

49 of 61 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personas28 Jun
  2. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey14 Jun

Other references · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
35 more
  1. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  2. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  3. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  4. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  5. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  6. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  7. 2024Phi-4 Technical Reportcited by 711
  8. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  9. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  10. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  11. 2023Textbooks Are All You Needcited by 569
  12. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  13. 2023Scaling Data-Constrained Language Modelscited by 403
  14. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  15. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  16. 2021Explaining Neural Scaling Lawscited by 363
  17. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  18. 2021Scaling Laws for Transfercited by 272
  19. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  20. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  21. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
  22. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  23. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 159
  24. 2023Self-Alignment with Instruction Backtranslationcited by 151
  25. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  26. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  27. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  28. 2024Common 7B Language Models Already Possess Strong Math Capabilitiescited by 109
  29. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  30. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 84
  31. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  32. 2025Distillation Scaling Lawscited by 67
  33. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 45
  34. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 39
  35. 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 26; 2026: 20.

46 papers.

Sort:

By area:

  1. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoningcited by 29
  2. 2025Synthetic Data RL: Task Definition Is All You Needcited by 15
  3. 2025MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence Calibrationcited by 11
  4. 2025Towards a Comprehensive Scaling Law of Mixture-of-Expertscited by 10
  5. 2025Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinkingcited by 10
  6. 2026Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewardscited by 10
  7. 2025Diversity-Enhanced Reasoning for Subjective Questionscited by 9
  8. 2026Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMscited by 9
  9. 2025Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformerscited by 8
  10. 2025Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forwardcited by 7
  11. 2025Merge-of-Thought Distillationcited by 7
  12. 2026Data-efficient pre-training by scaling synthetic megadocscited by 7
34 more
  1. 2026RebuttalAgent: Strategic Persuasion in Academic Rebuttal via Theory of Mindcited by 6
  2. 2026How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Datacited by 6
  3. 2025Mitigating Spurious Correlations in LLMs via Causality-Aware Post-Trainingcited by 5
  4. 2026Understanding Reasoning from Pretraining to Post-Trainingcited by 5
  5. 2025Thinking Augmented Pre-trainingcited by 4
  6. 2025More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoningcited by 4
  7. 2025Synthesizing Agentic Data for Web Agents with Progressive Difficulty Enhancement Mechanismscited by 4
  8. 2025QueST: Incentivizing LLMs to Generate Difficult Problemscited by 4
  9. 2026Training-Trajectory-Aware Token Selectioncited by 4
  10. 2026Scale Dependent Data Duplicationcited by 4
  11. 2025MAC-Tuning: LLM Multi-Compositional Problem Reasoning with Enhanced Knowledge Boundary Awarenesscited by 3
  12. 2025Scaling Laws for Code: A More Data-Hungry Regimecited by 3
  13. 2025Diffusion Models as Dataset Distillation Priorscited by 3
  14. 2026AgenticQwen: Training Small Agentic Language Models with Dual Data Flywheels for Industrial-Scale Tool Usecited by 3
  15. 2025Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Modelscited by 2
  16. 2025LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Pointscited by 2
  17. 2025Evaluating the Robustness of Chinchilla Compute-Optimal Scalingcited by 2
  18. 2025BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languagescited by 2
  19. 2025What Scales in Cross-Entropy Scaling Law?cited by 1
  20. 2026An Alternative Trajectory for Generative AIcited by 1
  21. 2026Eliciting Medical Reasoning with Knowledge-enhanced Data Synthesis: A Semi-Supervised Reinforcement Learning Approachcited by 1
  22. 2026CDR-Bench: Evaluating Faithful Execution of Compositional, Order-Sensitive Data Refinement Recipescited by 1
  23. 2025X-Intelligence 3.0: Training and Evaluating Reasoning LLM for Semiconductor Displaycited by 0
  24. 2025The Anti-Ouroboros Effect: Emergent Resilience in Large Language Models from Recursive Selective Feedbackcited by 0
  25. 2025SCoGen: Scenario-Centric Graph-Based Synthesis of Real-World Code Problemscited by 0
  26. 2025Transformers Can Learn Connectivity in Some Graphs but Not Otherscited by 0
  27. 2026From Atoms to Chains: Divergence-Guided Reasoning Curriculum for Unlabeled LLM Domain Adaptationcited by 0
  28. 2026R1-SyntheticVL: Is Synthetic Data from Generative Models Ready for Multimodal Large Language Model?cited by 0
  29. 2026VisionFoundry: Teaching VLMs Visual Perception with Synthetic Imagescited by 0
  30. 2026Scaling LLM Knowledge Boundaries via Distribution-Optimized Synthesiscited by 0
  31. 2026When Does Generating More Help? Disentangling Fixed-Source Synthesis from Source Expansion in Synthetic Data Scalingcited by 0
  32. 2026Clinical Communication Processing with Models Trained on LLM-Generated Synthetic Data: A Structured Survey and Novel Application Case Studiescited by 0
  33. 2026Learning with Synthetic Data via SGD in High-Dimensional Linear Regressioncited by 0
  34. 2026One Model Is Not a Crowd: Multi-LLM and Aspect-Conditioned Diverse Comment Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  2. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Models
  3. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  4. 2025BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining
  5. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  6. 2024Efficacy of Synthetic Data as a Benchmark
  7. 2025Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
  8. 2025Data Value in the Age of Scaling: Understanding LLM Scaling Dynamics Under Real-Synthetic Data Mixtures