mlsanityData to 7 Oct 2026

From Collapse to Improvement: Statistical Perspectives on the Evolutionary Dynamics of Iterative Training on Contaminated Sources

arXiv 2602.10531 ↗First posted 11 Feb 2026stat.ML cs.LG math.STcited by 0

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data

Abstract (arXiv)

The problem of model collapse has presented new challenges in iterative training of generative models, where such training with synthetic data leads to an overall degradation of performance. This paper looks at the problem from a statistical viewpoint, illustrating that one can actually hope for improvement when models are trained on data contaminated with synthetic samples, as long as there is some amount of fresh information from the true target distribution. In particular, we consider iterative training on samples sourced from a mixture of the true target and synthetic distributions. We analyze the entire iterative evolution in a next-token prediction language model, capturing how the interplay between the mixture weights and the sample size controls the overall long-term performance. With non-trivial mixture weight of the true distribution, even if it decays over time, simply training the model in a contamination-agnostic manner with appropriate sample sizes can avoid collapse and even recover the true target distribution under certain conditions. Simulation studies support our findings and also show that such behavior is more general for other classes of models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 26 of its 42 references that are in the map. First posted on arXiv 11 Feb 2026.

Builds on

26 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse7 Apr
  2. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data1 Apr
  3. 2024Model Collapse Demystified: The Case of Regression12 Feb
  4. 2024A Tale of Tails: Model Collapse as a Change of Scaling Laws10 Feb
  5. 2023The Curse of Recursion: Training on Generated Data Makes Models Forget27 May

Other references · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  7. 2016Density estimation using Real NVPcited by 2,029
  8. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  9. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  10. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  11. 2023Self-Consuming Generative Models Go MADcited by 239
  12. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 98
9 more
  1. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 97
  2. 2023Towards Understanding the Interplay of Generative Artificial Intelligence and the Internetcited by 54
  3. 2023Combining Generative Artificial Intelligence (AI) and the Internet: Heading towards Evolution or Degradation?cited by 41
  4. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 35
  5. 2024Towards Theoretical Understandings of Self-Consuming Generative Modelscited by 28
  6. 2023Nepotistically Trained Generative-AI Models Collapsecited by 27
  7. 2024The Rise of AI-Generated Content in Wikipediacited by 22
  8. 2024Regurgitative Training: The Value of Real Data in Training Large Language Modelscited by 15
  9. 2025A Training-free Method for LLM Text Attributioncited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025When Models Don't Collapse: On the Consistency of Iterative MLE
  2. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
  3. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World
  4. 2025Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework
  5. 2025A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops
  6. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective
  7. 2024Analyzing and Mitigating Model Collapse in Rectified Flow Models
  8. 2026Model Collapse as Cultural Evolution