mlsanityData to 7 Oct 2026

A Tale of Tails: Model Collapse as a Change of Scaling Laws

arXiv 2402.07043 ↗First posted 10 Feb 2024cs.LG cs.AI cs.CLcited by 127

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data

Abstract (arXiv)

As AI model size grows, neural scaling laws have become a crucial tool to predict the improvements of large models when increasing capacity and the size of original (human or natural) training data. Yet, the widespread use of popular models means that the ecosystem of online data and text will co-evolve to progressively contain increased amounts of synthesized data. In this paper we ask: How will the scaling laws change in the inevitable regime where synthetic data makes its way into the training corpus? Will future models, still improve, or be doomed to degenerate up to total (model) collapse? We develop a theoretical framework of model collapse through the lens of scaling laws. We discover a wide range of decay phenomena, analyzing loss of scaling, shifted scaling with number of generations, the ''un-learning" of skills, and grokking when mixing human and synthesized data. Our theory is validated by large-scale experiments with a transformer on an arithmetic task and text generation using the large language model Llama2.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 48 of its 68 references that are in the map. First posted on arXiv 10 Feb 2024.

Builds on

48 of 68 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023The Curse of Recursion: Training on Generated Data Makes Models Forget27 May

Other references · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
35 more
  1. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  2. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  3. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  4. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  5. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  6. 2021Scaling Laws for Transfercited by 265
  7. 2023Self-Consuming Generative Models Go MADcited by 239
  8. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  9. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  10. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  11. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  12. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  13. 2023The Quantization Model of Neural Scalingcited by 137
  14. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  15. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  16. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  17. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
  18. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 102
  19. 2022A Solvable Model of Neural Scaling Lawscited by 101
  20. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 98
  21. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 97
  22. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  23. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  24. 2023Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusioncited by 81
  25. 2021Learning Curve Theorycited by 79
  26. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  27. 2022Will Large-scale Generative Models Corrupt Future Datasets?cited by 59
  28. 2023Towards Understanding the Interplay of Generative Artificial Intelligence and the Internetcited by 54
  29. 2023Combining Generative Artificial Intelligence (AI) and the Internet: Heading towards Evolution or Degradation?cited by 41
  30. 2023Explore the Power of Synthetic Data on Few-shot Object Detectioncited by 38
  31. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
  32. 2022Evaluating Distributional Distortion in Neural Language Modelingcited by 18
  33. 2022Error Scaling Laws for Kernel Classification under Source and Capacity Conditionscited by 17
  34. 2023Image retrieval outperforms diffusion models on data augmentationcited by 14
  35. 2023Multiperiodic Processes: Ergodic Sources with a Sublinear Entropycited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 42; 2025: 46; 2026: 39.

127 papers.

Sort:

By area:

  1. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  2. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  3. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
  4. 2024Understanding Hallucinations in Diffusion Models through Mode Interpolationcited by 87
  5. 2024Model Collapse Demystified: The Case of Regressioncited by 80
  6. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapsecited by 75
  7. 2024Strong Model Collapsecited by 62
  8. 2024Large Language Models Can Self-Improve At Web Agent Taskscited by 54
  9. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54
  10. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 49
  11. 2025Diverse Preference Optimizationcited by 48
  12. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 43
38 more
  1. 2024Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Imagescited by 41
  2. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 40
  3. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 35
  4. 2024Scaling Optimal LR Across Token Horizonscited by 35
  5. 2024AI and the Problem of Knowledge Collapsecited by 34
  6. 2024Self-Improving Diffusion Models with Synthetic Datacited by 34
  7. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 32
  8. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 30
  9. 2024Universality of the π2/6 Pathway in Avoiding Model Collapsecited by 24
  10. 2024How to Synthesize Text Data without Model Collapse?cited by 24
  11. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfallscited by 24
  12. 2024Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agentscited by 22
  13. 2024Rate of Model Collapse in Recursive Trainingcited by 22
  14. 2024Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learningcited by 21
  15. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  16. 2024When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settingscited by 17
  17. 2025Improving the Scaling Laws of Synthetic Data with Deliberate Practicecited by 17
  18. 2024Loss-to-Loss Prediction: Scaling Laws for All Datasetscited by 15
  19. 2025A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loopscited by 15
  20. 2025Machine-generated text detection prevents language model collapsecited by 14
  21. 2024Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Samplingcited by 13
  22. 2024Bias Amplification: Large Language Models as Increasingly Biased Mediacited by 12
  23. 2025LLM as a Broken Telephone: Iterative Generation Distorts Informationcited by 12
  24. 2024Model Collapse in the Self-Consuming Chain of Diffusion Finetuning: A Novel Perspective from Quantitative Trait Modelingcited by 11
  25. 2024Emergent properties with repeated examplescited by 11
  26. 2025Escaping Collapse: The Strength of Weak Data for Large Language Model Trainingcited by 11
  27. 2025When Models Don't Collapse: On the Consistency of Iterative MLEcited by 11
  28. 2025Towards Lifecycle Unlearning Commitment Management: Measuring Sample-level Unlearning Completenesscited by 11
  29. 2024Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectracited by 9
  30. 2025Reformulation for Pretraining Data Augmentationcited by 9
  31. 2026R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Trainingcited by 9
  32. 2024Multi-Agent Sampling: Scaling Inference Compute for Data Synthesis with Tree Search-Based Agentic Collaborationcited by 8
  33. 2025Recursive Learning Without Collapse: A Weighting-Based Stabilization Frameworkcited by 8
  34. 2025Why Less is More (Sometimes): A Theory of Data Curationcited by 8
  35. 2024AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Lawscited by 7
  36. 2025Improved Scaling Laws in Linear Regression via Data Reusecited by 7
  37. 2025Riemannian-Geometric Fingerprints of Generative Modelscited by 7
  38. 2025Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regressioncited by 7

50 most cited of 127 All 127 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
  2. 2026Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data
  3. 2025How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines
  4. 2024Observational Scaling Laws and the Predictability of Language Model Performance
  5. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'
  6. 2024Scaling Laws for Pre-training Agents and World Models
  7. 2026Human-AI Co-Evolution and Epistemic Collapse: A Dynamical Systems Perspective
  8. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs