mlsanityData to 14 Oct 2026

Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory

arXiv 2410.08942 ↗First posted 11 Oct 2024cs.LG cs.AI math.STcited by 7

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Synthetic data has gained attention for training large language models, but poor-quality data can harm performance (see, e.g., Shumailov et al. (2023); Seddik et al. (2024)). A potential solution is data pruning, which retains only high-quality data based on a score function (human or machine feedback). Previous work Feng et al. (2024) analyzed models trained on synthetic data as sample size increases. We extend this by using random matrix theory to derive the performance of a binary classifier trained on a mix of real and pruned synthetic data in a high dimensional setting. Our findings identify conditions where synthetic data could improve performance, focusing on the quality of the generative model and verification strategy. We also show a smooth phase transition in synthetic label noise, contrasting with prior sharp behavior in infinite sample limits. Experiments with toy models and large language models validate our theoretical results.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 14 of its 22 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

14 of 22 references.

References · 14

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Qwen2 Technical Reportcited by 2,120
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  5. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  6. 2023Self-Consuming Generative Models Go MADcited by 250
  7. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 151
  8. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 100
  9. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 98
  10. 2024Model Collapse Demystified: The Case of Regressioncited by 82
  11. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapsecited by 79
  12. 2022Evaluating Distributional Distortion in Neural Language Modelingcited by 19
2 more
  1. 2024Falcon2-11B Technical Reportcited by 12
  2. 2024Alignment with Preference Optimization Is All You Need for LLM Safetycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025Improving the Scaling Laws of Synthetic Data with Deliberate Practicecited by 17
  2. 2025Escaping Collapse: The Strength of Weak Data for Large Language Model Trainingcited by 11
  3. 2025Why Less is More (Sometimes): A Theory of Data Curationcited by 8
  4. 2026Learning from Synthetic Data: Limitations of ERMcited by 3
  5. 2025Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Bootstrappingcited by 2
  6. 2025A Random Matrix Theory Perspective on the Learning Dynamics of Multi-head Latent Attentioncited by 1
  7. 2026Learning with Synthetic Data via SGD in High-Dimensional Linear Regressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Models
  2. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
  3. 2025Boosting Statistic Learning with Synthetic Data from Pretrained Large Models
  4. 2025Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
  5. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  6. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
  7. 2025Less is More: Adaptive Coverage for Synthetic Training Data
  8. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls