mlsanityData to 7 Oct 2026

Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

arXiv 2404.01413 ↗First posted 1 Apr 2024cs.LG cs.AI cs.CLcited by 143

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data

Abstract (arXiv)

The proliferation of generative models, combined with pretraining on web-scale data, raises a timely question: what happens when these models are trained on their own generated outputs? Recent investigations into model-data feedback loops proposed that such loops would lead to a phenomenon termed model collapse, under which performance progressively degrades with each model-data feedback iteration until fitted models become useless. However, those studies largely assumed that new data replace old data over time, where an arguably more realistic assumption is that data accumulate over time. In this paper, we ask: what effect does accumulating data have on model collapse? We empirically study this question by pretraining sequences of language models on text corpora. We confirm that replacing the original real data by each generation's synthetic data does indeed tend towards model collapse, then demonstrate that accumulating the successive generations of synthetic data alongside the original real data avoids model collapse; these results hold across a range of model sizes, architectures, and hyperparameters. We obtain similar results for deep generative models on other types of real data: diffusion models for molecule conformation generation and variational autoencoders for image generation. To understand why accumulating data can avoid model collapse, we use an analytically tractable framework introduced by prior work in which a sequence of linear models are fit to the previous models' outputs. Previous work used this framework to show that if data are replaced, the test error increases with the number of model-fitting iterations; we extend this argument to prove that if data instead accumulate, the test error has a finite upper bound independent of the number of iterations, meaning model collapse no longer occurs.

Key facts. Cited by 143 papers in the map (data through 7 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 24 of its 30 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

24 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Model Collapse Demystified: The Case of Regression12 Feb
  2. 2023The Curse of Recursion: Training on Generated Data Makes Models Forget27 May

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 511
  12. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
10 more
  1. 2023Self-Consuming Generative Models Go MADcited by 239
  2. 2020GEOM: Energy-annotated molecular conformations for property prediction and molecular generationcited by 238
  3. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 98
  4. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  5. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  6. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  7. 2022Data Feedback Loops: Model-driven Amplification of Dataset Biasescited by 59
  8. 2022Will Large-scale Generative Models Corrupt Future Datasets?cited by 59
  9. 2023Towards Understanding the Interplay of Generative Artificial Intelligence and the Internetcited by 54
  10. 2023Combining Generative Artificial Intelligence (AI) and the Internet: Heading towards Evolution or Degradation?cited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 55; 2026: 57.

143 papers.

Sort:

By area:

  1. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
  2. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  3. 2024Understanding Hallucinations in Diffusion Models through Mode Interpolationcited by 87
  4. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  5. 2025Reinforcement Learning from Human Feedbackcited by 53
  6. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 49
  7. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  8. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 40
  9. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 39
  10. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 36
  11. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 35
  12. 2024Self-Improving Diffusion Models with Synthetic Datacited by 34
38 more
  1. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 32
  2. 2025Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Modelscited by 31
  3. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 30
  4. 2025textttComplex-Edit: CoT-Like Instruction Generation for Complexity-Controllable Image Editing Benchmarkcited by 25
  5. 2024Universality of the π2/6 Pathway in Avoiding Model Collapsecited by 24
  6. 2024How to Synthesize Text Data without Model Collapse?cited by 24
  7. 2025Pre-training under infinite computecited by 24
  8. 2024Rate of Model Collapse in Recursive Trainingcited by 22
  9. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimizationcited by 20
  10. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  11. 2024When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settingscited by 17
  12. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 16
  13. 2026RAGEN-2: Reasoning Collapse in Agentic RLcited by 16
  14. 2024Regurgitative Training: The Value of Real Data in Training Large Language Modelscited by 15
  15. 2025A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loopscited by 15
  16. 2025Machine-generated text detection prevents language model collapsecited by 14
  17. 2025Efficient Prediction of Pass@k Scaling in Large Language Modelscited by 14
  18. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMscited by 12
  19. 2024Model Collapse in the Self-Consuming Chain of Diffusion Finetuning: A Novel Perspective from Quantitative Trait Modelingcited by 11
  20. 2025Escaping Collapse: The Strength of Weak Data for Large Language Model Trainingcited by 11
  21. 2025When Models Don't Collapse: On the Consistency of Iterative MLEcited by 11
  22. 2024Analyzing and Mitigating Model Collapse in Rectified Flow Modelscited by 10
  23. 2025Think, Prune, Train, Improve: Scaling Reasoning without Scaling Modelscited by 9
  24. 2025Value Drifts: Tracing Value Alignment During LLM Post-Trainingcited by 9
  25. 2024From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Modelscited by 8
  26. 2024Rephrasing natural text data with different languages and quality levels for Large Language Model pre-trainingcited by 8
  27. 2025Recursive Learning Without Collapse: A Weighting-Based Stabilization Frameworkcited by 8
  28. 2025Self-Consuming Generative Models with Adversarially Curated Datacited by 8
  29. 2025Why Less is More (Sometimes): A Theory of Data Curationcited by 8
  30. 2024Self-Directed Synthetic Dialogues and Revisions Technical Reportcited by 7
  31. 2025Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Trackcited by 7
  32. 2025Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regressioncited by 7
  33. 2024Multilingual Arbitrage: Optimizing Data Pools to Accelerate Multilingual Progresscited by 6
  34. 2024Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theorycited by 6
  35. 2025Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generationcited by 6
  36. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
  37. 2025RePro: Training Language Models to Faithfully Recycle the Web for Pretrainingcited by 6
  38. 2026The Impact of AI-Generated Text on the Internetcited by 6

50 most cited of 143 All 143 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse
  2. 2024A Tale of Tails: Model Collapse as a Change of Scaling Laws
  3. 2024Strong Model Collapse
  4. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification
  5. 2026When Does Model Collapse Occur in Structured Interactive Learning?
  6. 2026The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data
  7. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'
  8. 2025A Probabilistic Perspective on Model Collapse