mlsanityData to 7 Oct 2026

Model Collapse Demystified: The Case of Regression

arXiv 2402.07712 ↗First posted 12 Feb 2024cs.LG cs.AI stat.MLcited by 80

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data

Abstract (arXiv)

In the era of proliferation of large language and image generation models, the phenomenon of "model collapse" refers to the situation whereby as a model is trained recursively on data generated from previous generations of itself over time, its performance degrades until the model eventually becomes completely useless, i.e the model collapses. In this work, we study this phenomenon in the setting of high-dimensional regression and obtain analytic formulae which quantitatively outline this phenomenon in a broad range of regimes. In the special case of polynomial decaying spectral and source conditions, we obtain modified scaling laws which exhibit new crossover phenomena from fast to slow rates. We also propose a simple strategy based on adaptive regularization to mitigate model collapse. Our theoretical results are validated with experiments.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 41 of its 57 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

41 of 57 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024A Tale of Tails: Model Collapse as a Change of Scaling Laws10 Feb
  2. 2023The Curse of Recursion: Training on Generated Data Makes Models Forget27 May

Other references · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  11. 2018On Lazy Training in Differentiable Programmingcited by 671
  12. 2017Deep Neural Networks as Gaussian Processescited by 617
27 more
  1. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  2. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  3. 2023Self-Consuming Generative Models Go MADcited by 239
  4. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  5. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  6. 2018To understand deep learning we need to understand kernel learningcited by 186
  7. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  8. 2016Generalization Properties of Learning with Random Featurescited by 163
  9. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  10. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
  11. 2022A Solvable Model of Neural Scaling Lawscited by 101
  12. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 98
  13. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  14. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  15. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  16. 2018Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passescited by 69
  17. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 67
  18. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 61
  19. 2022Data Feedback Loops: Model-driven Amplification of Dataset Biasescited by 59
  20. 2022Will Large-scale Generative Models Corrupt Future Datasets?cited by 59
  21. 2023Towards Understanding the Interplay of Generative Artificial Intelligence and the Internetcited by 54
  22. 2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
  23. 2023Combining Generative Artificial Intelligence (AI) and the Internet: Heading towards Evolution or Degradation?cited by 41
  24. 2020Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Modelcited by 39
  25. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
  26. 2024Self-Correcting Self-Consuming Loops for Generative Model Trainingcited by 34
  27. 2022Error Scaling Laws for Kernel Classification under Source and Capacity Conditionscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 28; 2026: 30.

80 papers.

Sort:

By area:

  1. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 143
  2. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  3. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapsecited by 75
  4. 2024Strong Model Collapsecited by 62
  5. 2024LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large Multimodal Modelscited by 51
  6. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 49
  7. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 43
  8. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 35
  9. 2024Self-Improving Diffusion Models with Synthetic Datacited by 34
  10. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 32
  11. 2024Universality of the π2/6 Pathway in Avoiding Model Collapsecited by 24
  12. 2024How to Synthesize Text Data without Model Collapse?cited by 24
38 more
  1. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  2. 2024Rate of Model Collapse in Recursive Trainingcited by 22
  3. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  4. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 16
  5. 2025A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loopscited by 15
  6. 2024Bias Amplification: Large Language Models as Increasingly Biased Mediacited by 12
  7. 2025Ultra-Resolution Adaptation with Easecited by 12
  8. 2024Model Collapse in the Self-Consuming Chain of Diffusion Finetuning: A Novel Perspective from Quantitative Trait Modelingcited by 11
  9. 2025Escaping Collapse: The Strength of Weak Data for Large Language Model Trainingcited by 11
  10. 2025When Models Don't Collapse: On the Consistency of Iterative MLEcited by 11
  11. 2024Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectracited by 9
  12. 2025Recursive Learning Without Collapse: A Weighting-Based Stabilization Frameworkcited by 8
  13. 2025Why Less is More (Sometimes): A Theory of Data Curationcited by 8
  14. 2025On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspectivecited by 7
  15. 2025Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Trackcited by 7
  16. 2025Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regressioncited by 7
  17. 2024LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectivescited by 6
  18. 2024Multilingual Arbitrage: Optimizing Data Pools to Accelerate Multilingual Progresscited by 6
  19. 2024Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theorycited by 6
  20. 2026The Impact of AI-Generated Text on the Internetcited by 6
  21. 2025What happens when generative AI models train recursively on each others' outputs?cited by 5
  22. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergencecited by 5
  23. 2026Language Generation with Replay: A Learning-Theoretic View of Model Collapsecited by 5
  24. 2024Theoretical Proof that Auto-regressive Language Models Collapse when Real-world Data is a Finite Setcited by 4
  25. 2025Data Can Speak for Itself: Quality-guided Utilization of Wireless Synthetic Datacited by 4
  26. 2026Student Guides Teacher: Weak-to-Strong Inference via Spectral Orthogonal Explorationcited by 3
  27. 2026Quantifying Error Propagation and Model Collapse in Diffusion Modelscited by 3
  28. 2024Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Modelscited by 2
  29. 2025Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Bootstrappingcited by 2
  30. 2025Recursive Training Loops in LLMs: How training data properties modulate distribution shift in generated data?cited by 2
  31. 2025Privacy Amplification Through Synthetic Data: Insights from Linear Regressioncited by 2
  32. 2025Beyond Real Data: Synthetic Data through the Lens of Regularizationcited by 2
  33. 2025Diffusion Models as Dataset Distillation Priorscited by 2
  34. 2025The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curationcited by 2
  35. 2026Interestingness as an Inductive Heuristic for Future Compression Progresscited by 2
  36. 2026Self-Reference in Large Language Models: The Introspection Threshold for Recursive Self-Improvementcited by 2
  37. 2025Convergence Dynamics and Stabilization Strategies of Co-Evolving Generative Modelscited by 1
  38. 2025Turing Test 2.0: The General Intelligence Thresholdcited by 1

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Analyzing and Mitigating Model Collapse in Rectified Flow Models
  2. 2026Context Collapse: In-Context Learning and Model Collapse
  3. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'
  4. 2025It's Never Too Late: Noise Optimization for Collapse Recovery in Trained Diffusion Models
  5. 2025A Probabilistic Perspective on Model Collapse
  6. 2024The Prevalence of Neural Collapse in Neural Multivariate Regression
  7. 2025Stabilizing Self-Consuming Diffusion Models with Latent Space Filtering
  8. 2025A solvable model of learning generative diffusion: theory and insights