mlsanityData to 14 Oct 2026

On the Diversity of Synthetic Data and its Impact on Training Large Language Models

arXiv 2410.15226 ↗First posted 19 Oct 2024cs.CLcited by 59

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

The rise of Large Language Models (LLMs) has accentuated the need for diverse, high-quality pre-training data. Synthetic data emerges as a viable solution to the challenges of data scarcity and inaccessibility. While previous literature has focused predominantly on the quality and quantity of real data, our work enables the measurement of diversity in synthetic data and explores its impact on LLM performance. We study the downstream effects of synthetic data diversity during both the pre-training and fine-tuning stages by introducing a new diversity metric, textitLLM cluster-agent, designed to evaluate the diversity of synthetic datasets. Through a series of controlled experiments with models of 350M and 1.4B parameters, we demonstrate that the proposed cluster-based LLM scoring of diversity correlates positively with both pre-training and supervised fine-tuning performance. Our findings also reveal that synthetic data diversity in pre-training affects supervised fine-tuning more significantly than pre-training itself, even for smaller models. We hope this study advances our understanding of the optimal use of synthetic data in LLM training and opens new avenues for efficient data generation processes.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 68 of its 115 references that are in the map. First posted on arXiv 19 Oct 2024.

Builds on

68 of 115 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personas28 Jun
  2. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey14 Jun
  3. 2024Best Practices and Lessons Learned on Synthetic Data11 Apr

Other references · 65

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2023Mistral 7Bcited by 3,164
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  12. 2022Emergent Abilities of Large Language Modelscited by 2,512
38 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  2. 2024Qwen2 Technical Reportcited by 2,120
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  4. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  6. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  7. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  8. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  9. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  10. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  11. 2024Yi: Open Foundation Models by 01.AIcited by 719
  12. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  13. 2022Large Language Models Can Self-Improvecited by 664
  14. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  15. 2023Textbooks Are All You Needcited by 569
  16. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  17. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  18. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  19. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  20. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  21. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  22. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  23. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 211
  24. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  25. 2023Does Writing with Language Models Reduce Content Diversity?cited by 192
  26. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  27. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
  28. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
  29. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 143
  30. 2023What's In My Big Data?cited by 129
  31. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 108
  32. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 101
  33. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  34. 2024How to Train Data-Efficient LLMscited by 93
  35. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 84
  36. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 70
  37. 2019Jointly Measuring Diversity and Quality in Text Generation Modelscited by 61
  38. 2024Comprehensive Exploration of Synthetic Data Generation: A Surveycited by 60

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 36; 2026: 20.

59 papers.

Sort:

By area:

  1. 2025AgentEvolver: Towards Efficient Self-Evolving Agent Systemcited by 85
  2. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 84
  3. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  4. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 43
  5. 2025Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoningcited by 40
  6. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  7. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfallscited by 26
  8. 2025Rewriting Pre-Training Data Boosts LLM Performance in Math and Codecited by 21
  9. 2025General Intelligence Requires Reward-based Pretrainingcited by 12
  10. 2025BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generationcited by 8
  11. 2025Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levelscited by 8
  12. 2025Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformerscited by 8
38 more
  1. 2026DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Modelscited by 8
  2. 2025What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversitycited by 7
  3. 2025RePro: Training Language Models to Faithfully Recycle the Web for Pretrainingcited by 6
  4. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergencecited by 6
  5. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 6
  6. 2025LLM Unlearning Without an Expert Curated Datasetcited by 5
  7. 2026Mi:dm 2.0 Korea-centric Bilingual Language Modelscited by 5
  8. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  9. 2026Reasoning-Driven Synthetic Data Generation and Evaluationcited by 5
  10. 2025Out-of-Distribution Detection using Synthetic Data Generationcited by 4
  11. 2025A Survey of LLM × DATAcited by 4
  12. 2025KORMo: Korean Open Reasoning Model for Everyonecited by 4
  13. 2026Scale Dependent Data Duplicationcited by 4
  14. 2025From Macro to Micro: Probing Dataset Diversity in Language Model Fine-Tuningcited by 3
  15. 2025Synthetic Voice Data for Automatic Speech Recognition in African Languagescited by 3
  16. 2025MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sourcescited by 3
  17. 2026A Sustainable AI Economy Needs Data Deals That Work for Generatorscited by 3
  18. 2026Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillationcited by 3
  19. 2025Modelling and Classifying the Components of a Literature Reviewcited by 2
  20. 2025Exploring Solution Divergence and Its Effect on Large Language Model Problem Solvingcited by 2
  21. 2026Generating Pretraining Tokens from Organic Data for Data-Bound Scalingcited by 2
  22. 2026Mechanistic Circuit Identification for Controllable Data Generationcited by 2
  23. 2024Instructional Text Across Disciplines: A Survey of Representations, Downstream Tasks, and Open Challenges Toward Capable AI Agentscited by 1
  24. 2025Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimizationcited by 1
  25. 2025Improving Multilingual Math Reasoning for African Languagescited by 1
  26. 2025What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuningcited by 1
  27. 2025Privacy-Preserving Synthetic Review Generation with Diverse Writing Styles Using LLMscited by 1
  28. 2025Verification Limits Code LLM Trainingcited by 1
  29. 2025AAPA: Adversarially Anchored Preference Alignment for Post-Training of Large Language Modelscited by 1
  30. 2025Data Value in the Age of Scaling: Understanding LLM Scaling Dynamics Under Real-Synthetic Data Mixturescited by 1
  31. 2026Team, Then Trim: An Assembly-Line LLM Framework for High-Quality Tabular Data Generationcited by 1
  32. 2026The Magic Correlations: Understanding Knowledge Transfer from Pretraining to Supervised Fine-Tuningcited by 1
  33. 2026Synthetic Data Generation for Training Diversified Commonsense Reasoning Modelscited by 1
  34. 2024"Moralized" Multi-Step Jailbreak Prompts: Black-Box Testing of Guardrails in Large Language Models for Verbal Attackscited by 0
  35. 2025Facts Do Care About Your Language: Assessing Answer Quality of Multilingual LLMscited by 0
  36. 2025Language Models Might Not Understand You: Evaluating Theory of Mind via Story Promptingcited by 0
  37. 2025Balancing Synthetic Data and Replay for Enhancing Task-Specific Capabilitiescited by 0
  38. 2025AutoSynth: Automated Workflow Optimization for High-Quality Synthetic Dataset Generation via Monte Carlo Tree Searchcited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning
  2. 2025Scaling Laws of Synthetic Data for Language Models
  3. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
  4. 2024Efficacy of Synthetic Data as a Benchmark
  5. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  6. 2024Hybrid Training Approaches for LLMs: Leveraging Real and Synthetic Data to Enhance Model Performance in Domain-Specific Applications
  7. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  8. 2025Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios