mlsanityData to 14 Oct 2026

When Models Don't Collapse: On the Consistency of Iterative MLE

arXiv 2505.19046 ↗First posted 25 May 2025stat.ML cs.LGcited by 13

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data

Abstract (arXiv)

The widespread use of generative models has created a feedback loop, in which each generation of models is trained on data partially produced by its predecessors. This process has raised concerns about model collapse: A critical degradation in performance caused by repeated training on synthetic data. However, different analyses in the literature have reached different conclusions as to the severity of model collapse. As such, it remains unclear how concerning this phenomenon is, and under which assumptions it can be avoided. To address this, we theoretically study model collapse for maximum likelihood estimation (MLE), in a natural setting where synthetic data is gradually added to the original data set. Under standard assumptions (similar to those long used for proving asymptotic consistency and normality of MLE), we establish non-asymptotic bounds showing that collapse can be avoided even as the fraction of real data vanishes. On the other hand, we prove that some assumptions (beyond MLE consistency) are indeed necessary: Without them, model collapse can occur arbitrarily quickly, even when the original data is still present in the training set. To the best of our knowledge, these are the first rigorous examples of iterative generative modeling with accumulating data that rapidly leads to model collapse.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 25 of its 43 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

25 of 43 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Position: Model Collapse Does Not Mean What You Think5 Mar
  2. 2025A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops26 Feb
  3. 2025Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework25 Feb
  4. 2024Strong Model Collapse7 Oct
  5. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification11 Jun
  6. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse7 Apr
  7. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data1 Apr
  8. 2024Model Collapse Demystified: The Case of Regression12 Feb
  9. 2024A Tale of Tails: Model Collapse as a Change of Scaling Laws10 Feb

Other references · 16

  1. 2023Self-Consuming Generative Models Go MADcited by 250
  2. 2019A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Normcited by 106
  3. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 100
  4. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 100
  5. 2021Learning Curve Theorycited by 85
  6. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 79
  7. 2022Will Large-scale Generative Models Corrupt Future Datasets?cited by 59
  8. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 52
  9. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 39
  10. 2023Scaling Laws for Associative Memoriescited by 35
  11. 2024Towards Theoretical Understandings of Self-Consuming Generative Modelscited by 29
  12. 2023Nepotistically Trained Generative-AI Models Collapsecited by 27
4 more
  1. 2024Scaling laws for learning with real and surrogate datacited by 27
  2. 2024Universality of the π2/6 Pathway in Avoiding Model Collapsecited by 26
  3. 2024Heat Death of Generative Models in Closed-Loop Learningcited by 22
  4. 2025Escaping Collapse: The Strength of Weak Data for Large Language Model Trainingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 10.

13 papers.

Sort:

By area:

  1. 2025What happens when generative AI models train recursively on each others' outputs?cited by 6
  2. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergencecited by 6
  3. 2026Language Generation with Replay: A Learning-Theoretic View of Model Collapsecited by 5
  4. 2026Quantifying Error Propagation and Model Collapse in Diffusion Modelscited by 4
  5. 2025Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Promptingcited by 3
  6. 2026Learning from Synthetic Data: Limitations of ERMcited by 3
  7. 2026When Does Model Collapse Occur in Structured Interactive Learning?cited by 2
  8. 2026Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Trainingcited by 0
  9. 2026Recursively Trained Diffusion Models: Limiting Collapse Distribution and Spectral Characterizationcited by 0
  10. 2026Minimax bounds for watermarked and masked recursive discrete distribution estimationcited by 0
  11. 2026Learning with Synthetic Data via SGD in High-Dimensional Linear Regressioncited by 0
  12. 2026Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Trainingcited by 0
1 more
  1. 2026Screening Collapse under Recursive Synthetic Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'
  2. 2026From Collapse to Improvement: Statistical Perspectives on the Evolutionary Dynamics of Iterative Training on Contaminated Sources
  3. 2026Reviewing Model Collapse and Countermeasures
  4. 2026Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data
  5. 2024Rate of Model Collapse in Recursive Training
  6. 2025ForTIFAI: Fending Off Recursive Training Induced Failure for AI Model Collapse
  7. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective
  8. 2024Analyzing and Mitigating Model Collapse in Rectified Flow Models