mlsanityData to 7 Oct 2026

Strong Model Collapse

arXiv 2410.04840 ↗First posted 7 Oct 2024cs.LG stat.MLcited by 62

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data

Abstract (arXiv)

Within the scaling laws paradigm, which underpins the training of large neural networks like ChatGPT and Llama, we consider a supervised regression setting and establish the existance of a strong form of the model collapse phenomenon, a critical performance degradation due to synthetic data in the training corpus. Our results show that even the smallest fraction of synthetic data (e.g., as little as 1% of the total training dataset) can still lead to model collapse: larger and larger training sets do not enhance performance. We further investigate whether increasing model size, an approach aligned with current trends in training large language models, exacerbates or mitigates model collapse. In a simplified regime where neural networks are approximated via random projections of tunable size, we both theoretically and empirically show that larger models can amplify model collapse. Interestingly, our theory also indicates that, beyond the interpolation threshold (which can be extremely high for very large datasets), larger models may mitigate the collapse, although they do not entirely prevent it. Our theoretical findings are empirically verified through experiments on language models and feed-forward neural networks for images.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 36 of its 53 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

36 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse7 Apr
  2. 2024Model Collapse Demystified: The Case of Regression12 Feb
  3. 2024A Tale of Tails: Model Collapse as a Change of Scaling Laws10 Feb
  4. 2023The Curse of Recursion: Training on Generated Data Makes Models Forget27 May

Other references · 32

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2024Mixtral of Expertscited by 1,785
  6. 2018On Lazy Training in Differentiable Programmingcited by 671
  7. 2017Deep Neural Networks as Gaussian Processescited by 617
  8. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  9. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  10. 2023Self-Consuming Generative Models Go MADcited by 239
  11. 2016Generalization Properties of Learning with Random Featurescited by 163
  12. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
20 more
  1. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  2. 2022A Solvable Model of Neural Scaling Lawscited by 101
  3. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 98
  4. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  5. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  6. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 61
  7. 2022Will Large-scale Generative Models Corrupt Future Datasets?cited by 59
  8. 2023Towards Understanding the Interplay of Generative Artificial Intelligence and the Internetcited by 54
  9. 2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
  10. 2023Combining Generative Artificial Intelligence (AI) and the Internet: Heading towards Evolution or Degradation?cited by 41
  11. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 35
  12. 2024Self-Correcting Self-Consuming Loops for Generative Model Trainingcited by 34
  13. 2024Self-Improving Diffusion Models with Synthetic Datacited by 34
  14. 2024Scaling laws for learning with real and surrogate datacited by 26
  15. 2021Covariate Shift in High-Dimensional Random Feature Regressioncited by 24
  16. 2022Error Scaling Laws for Kernel Classification under Source and Capacity Conditionscited by 17
  17. 2024Memory Mosaicscited by 16
  18. 2024Regurgitative Training: The Value of Real Data in Training Large Language Modelscited by 15
  19. 2023Demystifying Disagreement-on-the-Line in High Dimensionscited by 12
  20. 2020Weighted Empirical Risk Minimization: Sample Selection Bias Correction based on Importance Samplingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 31; 2026: 28.

62 papers.

Sort:

By area:

  1. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  2. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 32
  3. 2024How to Synthesize Text Data without Model Collapse?cited by 24
  4. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfallscited by 24
  5. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 22
  6. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  7. 2025Improving the Scaling Laws of Synthetic Data with Deliberate Practicecited by 17
  8. 2025A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loopscited by 15
  9. 2025Knowledge Updating? No More Model Editing! Just Selective Contextual Reasoningcited by 15
  10. 2025Ultra-Resolution Adaptation with Easecited by 12
  11. 2025When Models Don't Collapse: On the Consistency of Iterative MLEcited by 11
  12. 2024Analyzing and Mitigating Model Collapse in Rectified Flow Modelscited by 10
38 more
  1. 2025Reformulation for Pretraining Data Augmentationcited by 9
  2. 2025Recursive Learning Without Collapse: A Weighting-Based Stabilization Frameworkcited by 8
  3. 2025Why Less is More (Sometimes): A Theory of Data Curationcited by 8
  4. 2025Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMscited by 7
  5. 2025Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regressioncited by 7
  6. 2025RePro: Training Language Models to Faithfully Recycle the Web for Pretrainingcited by 6
  7. 2026The Impact of AI-Generated Text on the Internetcited by 6
  8. 2026Unlocking LLM Creativity in Science through Analogical Reasoningcited by 6
  9. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergencecited by 5
  10. 2026Language Generation with Replay: A Learning-Theoretic View of Model Collapsecited by 5
  11. 2024Asymptotics of Linear Regression with Linearly Dependent Datacited by 4
  12. 2025AI-Enhanced Deliberative Democracy and the Future of the Collective Willcited by 4
  13. 2025Generating on Generated: An Approach Towards Self-Evolving Diffusion Modelscited by 3
  14. 2026A Sustainable AI Economy Needs Data Deals That Work for Generatorscited by 3
  15. 2026Quantifying Error Propagation and Model Collapse in Diffusion Modelscited by 3
  16. 2025Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Bootstrappingcited by 2
  17. 2025Beyond Real Data: Synthetic Data through the Lens of Regularizationcited by 2
  18. 2025Uncertainty Quantification and Data Efficiency in AI: An Information-Theoretic Perspectivecited by 2
  19. 2026Preventing Curriculum Collapse in Self-Evolving Reasoning Systemscited by 2
  20. 2026Generating Pretraining Tokens from Organic Data for Data-Bound Scalingcited by 2
  21. 2025We Need Improved Data Curation and Attribution in AI for Scientific Discoverycited by 1
  22. 2025What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuningcited by 1
  23. 2025Knowledge Collapse in LLMs: When Fluency Survives but Facts Fail under Recursive Synthetic Trainingcited by 1
  24. 2025ForTIFAI: Fending Off Recursive Training Induced Failure for AI Model Collapsecited by 1
  25. 2025A Criminology of Machinescited by 1
  26. 2025Data Value in the Age of Scaling: Understanding LLM Scaling Dynamics Under Real-Synthetic Data Mixturescited by 1
  27. 2026Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communitiescited by 1
  28. 2026Model Collapse as Cultural Evolutioncited by 1
  29. 2025auto-fpt: Automating Free Probability Theory Calculations for Machine Learning Theorycited by 0
  30. 2025Benchmarking and Rethinking Knowledge Editing for Large Language Modelscited by 0
  31. 2025Future of AI Models: A Computational perspective on Model collapsecited by 0
  32. 2026SIGMA: Scalable Spectral Insights for LLM Model Collapsecited by 0
  33. 2026Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Trainingcited by 0
  34. 2026LLM Ghostbusters: Surgical Package Hallucination Suppression via Adaptive Unlearningcited by 0
  35. 2026Generative artificial intelligence and the marginalization of minoritized knowledges in higher education: the case of disabilitycited by 0
  36. 2026When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loopcited by 0
  37. 2026Entropy Minimization without Model Collapse: Mitigating Prediction Bias in Medical Imagingcited by 0
  38. 2026Epidemiology of Model Collapse: Modeling Synthetic Data Contamination via Bilayer SIR Dynamicscited by 0

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data
  2. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
  3. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification
  4. 2025A Probabilistic Perspective on Model Collapse
  5. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World
  6. 2025Machine-generated text detection prevents language model collapse
  7. 2024Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Models
  8. 2026Context Collapse: In-Context Learning and Model Collapse