mlsanityData to 7 Oct 2026

A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops

arXiv 2502.18865 ↗First posted 26 Feb 2025cs.LG cs.AIcited by 15

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data

Abstract (arXiv)

High-quality data is essential for training large generative models, yet the vast reservoir of real data available online has become nearly depleted. Consequently, models increasingly generate their own data for further training, forming Self-consuming Training Loops (STLs). However, the empirical results have been strikingly inconsistent: some models degrade or even collapse, while others successfully avoid these failures, leaving a significant gap in theoretical understanding to explain this discrepancy. This paper introduces the intriguing notion of recursive stability and presents the first theoretical generalization analysis, revealing how both model architecture and the proportion between real and synthetic data influence the success of STLs. We further extend this analysis to transformers in in-context learning, showing that even a constant-sized proportion of real data ensures convergence, while also providing insights into optimal synthetic data sizing.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 36 of its 52 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

36 of 52 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Strong Model Collapse7 Oct
  2. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification11 Jun
  3. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse7 Apr
  4. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data1 Apr
  5. 2024Model Collapse Demystified: The Case of Regression12 Feb
  6. 2024A Tale of Tails: Model Collapse as a Change of Scaling Laws10 Feb

Other references · 30

  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2022Large Language Models Can Self-Improvecited by 656
  3. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  4. 2023Can AI-Generated Text be Reliably Detected?cited by 372
  5. 2023Self-Consuming Generative Models Go MADcited by 239
  6. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  7. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 172
  8. 2020Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descentcited by 118
  9. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  10. 2017Stability and Generalization of Learning Algorithms that Converge to Global Optimacited by 100
  11. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 98
  12. 2019High probability generalization bounds for uniformly stable algorithms with nearly optimal ratecited by 97
18 more
  1. 2019Sharper bounds for uniformly stable algorithmscited by 91
  2. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  3. 2018How Well Generative Adversarial Networks Learn Distributionscited by 78
  4. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  5. 2024Fairness Feedback Loops: Training on Synthetic Data Amplifies Biascited by 55
  6. 2023Towards Understanding the Interplay of Generative Artificial Intelligence and the Internetcited by 54
  7. 2020Train simultaneously, generalize better: Stability of gradient-based minimax learnerscited by 43
  8. 2021Stability and Deviation Optimal Risk Bounds with Convergence Rate O(1/n)cited by 42
  9. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 35
  10. 2024Self-Correcting Self-Consuming Loops for Generative Model Trainingcited by 34
  11. 2024Self-Improving Diffusion Models with Synthetic Datacited by 34
  12. 2021Stability of SGD: Tightness Analysis and Improved Boundscited by 33
  13. 2024Towards Theoretical Understandings of Self-Consuming Generative Modelscited by 28
  14. 2022Stability and Generalization of Stochastic Optimization with Nonconvex and Nonsmooth Problemscited by 25
  15. 2023Toward Understanding Generative Data Augmentationcited by 25
  16. 2024Heat Death of Generative Models in Closed-Loop Learningcited by 21
  17. 2023Utility Theory of Synthetic Data Generationcited by 8
  18. 2022A Mathematical Framework for Learning Probability Distributionscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 10.

15 papers.

Sort:

By area:

  1. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  2. 2025When Models Don't Collapse: On the Consistency of Iterative MLEcited by 11
  3. 2026Self-Evolving Agents with Anytime-Valid Certificatescited by 6
  4. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergencecited by 5
  5. 2026Language Generation with Replay: A Learning-Theoretic View of Model Collapsecited by 5
  6. 2025Stabilizing Self-Consuming Diffusion Models with Latent Space Filteringcited by 3
  7. 2025Epistemic diversity across language models mitigates knowledge collapsecited by 2
  8. 2026Ambient Dataloops: Generative Models for Dataset Refinementcited by 2
  9. 2026When AI Reviews Its Own Code: Recursive Self-Training Collapse in Code LLMscited by 2
  10. 2026A Multi-Timescale Recursive Self-Improvement Engine for Open-Ended Persona Growthcited by 1
  11. 2026When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loopcited by 0
  12. 2026Minimax bounds for watermarked and masked recursive discrete distribution estimationcited by 0
3 more
  1. 2026SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Datacited by 0
  2. 2026Recursive LLM Degradation in Biomedical Question Answering: A Cross-Generation Studycited by 0
  3. 2026No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapsecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework
  2. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World
  3. 2026Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data
  4. 2025ForTIFAI: Fending Off Recursive Training Induced Failure for AI Model Collapse
  5. 2024Analyzing and Mitigating Model Collapse in Rectified Flow Models
  6. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'
  7. 2026From Collapse to Improvement: Statistical Perspectives on the Evolutionary Dynamics of Iterative Training on Contaminated Sources
  8. 2025A Probabilistic Perspective on Model Collapse