mlsanityData to 7 Oct 2026

How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse

arXiv 2404.05090 ↗First posted 7 Apr 2024cs.LG cs.AI cs.CLcited by 75

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data

Abstract (arXiv)

The phenomenon of model collapse, introduced in (Shumailov et al., 2023), refers to the deterioration in performance that occurs when new models are trained on synthetic data generated from previously trained models. This recursive training loop makes the tails of the original distribution disappear, thereby making future-generation models forget about the initial (real) distribution. With the aim of rigorously understanding model collapse in language models, we consider in this paper a statistical model that allows us to characterize the impact of various recursive training scenarios. Specifically, we demonstrate that model collapse cannot be avoided when training solely on synthetic data. However, when mixing both real and synthetic data, we provide an estimate of a maximal amount of synthetic data below which model collapse can eventually be avoided. Our theoretical conclusions are further supported by empirical validations.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 8 of its 16 references that are in the map. First posted on arXiv 7 Apr 2024.

Builds on

8 of 16 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Model Collapse Demystified: The Case of Regression12 Feb
  2. 2024A Tale of Tails: Model Collapse as a Change of Scaling Laws10 Feb
  3. 2023The Curse of Recursion: Training on Generated Data Makes Models Forget27 May

Other references · 5

  1. 2023Textbooks Are All You Needcited by 561
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  3. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  4. 2024Towards Theoretical Understandings of Self-Consuming Generative Modelscited by 28
  5. 2023Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflowcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 27; 2026: 31.

75 papers.

Sort:

By area:

  1. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  2. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  3. 2024Strong Model Collapsecited by 62
  4. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 49
  5. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 43
  6. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 32
  7. 2024Universality of the π2/6 Pathway in Avoiding Model Collapsecited by 24
  8. 2024How to Synthesize Text Data without Model Collapse?cited by 24
  9. 2024Rate of Model Collapse in Recursive Trainingcited by 22
  10. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  11. 2025Superficial Self-Improved Reasoners Benefit from Model Mergingcited by 16
  12. 2025A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loopscited by 15
38 more
  1. 2025When Your AI Agent Succumbs to Peer-Pressure: Studying Opinion-Change Dynamics of LLMscited by 14
  2. 2024Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Modelscited by 13
  3. 2025NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communitiescited by 13
  4. 2024Montessori-Instruct: Generate Influential Training Data Tailored for Student Learningcited by 12
  5. 2024Bias Amplification: Large Language Models as Increasingly Biased Mediacited by 12
  6. 2025Do we really have to filter out random noise in pre-training data for language models?cited by 11
  7. 2025Escaping Collapse: The Strength of Weak Data for Large Language Model Trainingcited by 11
  8. 2025When Models Don't Collapse: On the Consistency of Iterative MLEcited by 11
  9. 2025Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Modelscited by 9
  10. 2025Why Less is More (Sometimes): A Theory of Data Curationcited by 8
  11. 2024FLARE up your data: Diffusion-based Augmentation Method in Astronomical Imagingcited by 7
  12. 2024Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theorycited by 6
  13. 2024A Little Human Data Goes A Long Waycited by 6
  14. 2024DAWN-ICL: Strategic Planning of Problem-solving Trajectories for Zero-Shot In-Context Learningcited by 6
  15. 2025LLM Web Dynamics: Tracing Model Collapse in a Network of LLMscited by 6
  16. 2025Multi-modal Synthetic Data Training and Model Collapse: Insights from VLMs and Diffusion Modelscited by 5
  17. 2026Language Generation with Replay: A Learning-Theoretic View of Model Collapsecited by 5
  18. 2024Theoretical Proof that Auto-regressive Language Models Collapse when Real-world Data is a Finite Setcited by 4
  19. 2025Unlocking Speech Instruction Data Potential with Query Rewritingcited by 4
  20. 2025Does Training on Synthetic Data Make Models Less Robust?cited by 3
  21. 2025Using Sign Language Production as Data Augmentation to enhance Sign Language Translationcited by 3
  22. 2025Synthetic Voice Data for Automatic Speech Recognition in African Languagescited by 3
  23. 2026Student Guides Teacher: Weak-to-Strong Inference via Spectral Orthogonal Explorationcited by 3
  24. 2026Learning from Synthetic Data: Limitations of ERMcited by 3
  25. 2026Markovian Generation Chains in Large Language Modelscited by 3
  26. 2024Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Modelscited by 2
  27. 2025EnAnchored-X2X: English-Anchored Optimization for Many-to-Many Translationcited by 2
  28. 2025LLMs Can Get "Brain Rot": A Pilot Study on Twitter/Xcited by 2
  29. 2026Ambient Dataloops: Generative Models for Dataset Refinementcited by 2
  30. 2026Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillationcited by 2
  31. 2026When AI Reviews Its Own Code: Recursive Self-Training Collapse in Code LLMscited by 2
  32. 2025What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuningcited by 1
  33. 2025Knowledge Collapse in LLMs: When Fluency Survives but Facts Fail under Recursive Synthetic Trainingcited by 1
  34. 2025First-Extinction Law for Resampling Processescited by 1
  35. 2025Data Value in the Age of Scaling: Understanding LLM Scaling Dynamics Under Real-Synthetic Data Mixturescited by 1
  36. 2026Observations and Remedies for Large Language Model Bias in Self-Consuming Performative Loopcited by 1
  37. 2026Integrating Homomorphic Encryption and Synthetic Data in FL for Privacy and Learning Qualitycited by 1
  38. 2026Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communitiescited by 1

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Probabilistic Perspective on Model Collapse
  2. 2026Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data
  3. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
  4. 2025Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
  5. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
  6. 2025Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning
  7. 2025ForTIFAI: Fending Off Recursive Training Induced Failure for AI Model Collapse
  8. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls