mlsanityData to 14 Oct 2026

On the Stability of Iterative Retraining of Generative Models on their own Data

arXiv 2310.00429 ↗First posted 30 Sep 2023cs.LG stat.MLcited by 100

Not on the map. Why?

Cited by 42 in Synthetic data and self-improvement for LLMs, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep generative models have made tremendous progress in modeling complex data, often exhibiting generation quality that surpasses a typical human's ability to discern the authenticity of samples. Undeniably, a key driver of this success is enabled by the massive amounts of web-scale data consumed by these models. Due to these models' striking performance and ease of availability, the web will inevitably be increasingly populated with synthetic content. Such a fact directly implies that future iterations of generative models will be trained on both clean and artificially generated data from past models. In this paper, we develop a framework to rigorously study the impact of training generative models on mixed datasets -- from classical training on real data to self-consuming generative models trained on purely synthetic data. We first prove the stability of iterative training under the condition that the initial generative models approximate the data distribution well enough and the proportion of clean training data (w.r.t. synthetic data) is large enough. We empirically validate our theory on both synthetic and natural images by iteratively training normalizing flows and state-of-the-art diffusion models on CIFAR10 and FFHQ.

Key facts. Cited by 100 papers in the map (data through 14 Oct 2026). Builds on 38 of its 53 references that are in the map. First posted on arXiv 30 Sep 2023.

Builds on

38 of 53 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  7. 2022Flow Matching for Generative Modelingcited by 5,702
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2018Neural Ordinary Differential Equationscited by 3,779
  12. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
26 more
  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  2. 2021Convex optimizationcited by 2,515
  3. 2015Variational Inference with Normalizing Flowscited by 2,170
  4. 2023PaLM 2 Technical Reportcited by 1,162
  5. 2019An Introduction to Variational Autoencoderscited by 1,000
  6. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  7. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 879
  8. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 726
  9. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 594
  10. 2018Assessing Generative Models via Precision and Recallcited by 406
  11. 2023Can AI-Generated Text be Reliably Detected?cited by 386
  12. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  13. 2020Performative Predictioncited by 327
  14. 2017Data Augmentation Generative Adversarial Networkscited by 278
  15. 2023Self-Consuming Generative Models Go MADcited by 250
  16. 2023Diffusion Models are Minimax Optimal Distribution Estimatorscited by 165
  17. 2020Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximatorscited by 85
  18. 2022Will Large-scale Generative Models Corrupt Future Datasets?cited by 59
  19. 2023Towards Understanding the Interplay of Generative Artificial Intelligence and the Internetcited by 54
  20. 2018Generalization Error in Deep Learningcited by 47
  21. 2023Combining Generative Artificial Intelligence (AI) and the Internet: Heading towards Evolution or Degradation?cited by 41
  22. 2023Performative Prediction with Neural Networkscited by 34
  23. 2021A single gradient step finds adversarial examples on random two-layers neural networkscited by 26
  24. 2020Generalization and Memorization: The Bias Potential Modelcited by 9
  25. 2021Generalization Error of GAN from the Discriminator's Perspectivecited by 8
  26. 2020Multivariate Gaussian Variational Inference by Natural Gradient Descentcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 29; 2025: 33; 2026: 35.

100 papers.

Sort:

By area:

  1. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 151
  2. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 130
  3. 2024Iterated Denoising Energy Matching for Sampling from Boltzmann Densitiescited by 116
  4. 2024Understanding Hallucinations in Diffusion Models through Mode Interpolationcited by 91
  5. 2024Model Collapse Demystified: The Case of Regressioncited by 82
  6. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 79
  7. 2024Strong Model Collapsecited by 65
  8. 2023Performative Prediction: Past and Futurecited by 53
  9. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 52
  10. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 45
  11. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 39
  12. 2024Self-Correcting Self-Consuming Loops for Generative Model Trainingcited by 37
38 more
  1. 2024Self-Improving Diffusion Models with Synthetic Datacited by 35
  2. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 33
  3. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 31
  4. 2024Towards Theoretical Understandings of Self-Consuming Generative Modelscited by 29
  5. 2024Universality of the π2/6 Pathway in Avoiding Model Collapsecited by 26
  6. 2024How to Synthesize Text Data without Model Collapse?cited by 25
  7. 2024Rate of Model Collapse in Recursive Trainingcited by 25
  8. 2024Heat Death of Generative Models in Closed-Loop Learningcited by 22
  9. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 18
  10. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  11. 2025A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loopscited by 16
  12. 2025Superficial Self-Improved Reasoners Benefit from Model Mergingcited by 16
  13. 2024Regurgitative Training: The Value of Real Data in Training Large Language Modelscited by 15
  14. 2024Understanding Bias in Large-Scale Visual Datasetscited by 15
  15. 2025Machine-generated text detection prevents language model collapsecited by 14
  16. 2025When Models Don't Collapse: On the Consistency of Iterative MLEcited by 13
  17. 2024Model Collapse in the Self-Consuming Chain of Diffusion Finetuning: A Novel Perspective from Quantitative Trait Modelingcited by 12
  18. 2024Analyzing and Mitigating Model Collapse in Rectified Flow Modelscited by 11
  19. 2025Escaping Collapse: The Strength of Weak Data for Large Language Model Trainingcited by 11
  20. 2024Beware of Words: Evaluating the Lexical Diversity of Conversational LLMs using ChatGPT as Case Studycited by 10
  21. 2025Recursive Learning Without Collapse: A Weighting-Based Stabilization Frameworkcited by 10
  22. 2025Self-Consuming Generative Models with Adversarially Curated Datacited by 10
  23. 2024Automating Data Annotation under Strategic Human Agents: Risks and Potential Solutionscited by 9
  24. 2025Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regressioncited by 9
  25. 2025Why Less is More (Sometimes): A Theory of Data Curationcited by 8
  26. 2024Multilingual Arbitrage: Optimizing Data Pools to Accelerate Multilingual Progresscited by 7
  27. 2025Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMscited by 7
  28. 2024LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectivescited by 6
  29. 2025Iterative Importance Fine-tuning of Diffusion Modelscited by 6
  30. 2025LLM Web Dynamics: Tracing Model Collapse in a Network of LLMscited by 6
  31. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
  32. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergencecited by 6
  33. 2025Neon: Negative Extrapolation From Self-Training Improves Image Generationcited by 5
  34. 2026Language Generation with Replay: A Learning-Theoretic View of Model Collapsecited by 5
  35. 2024Data Diversity as Implicit Regularization: How Does Diversity Shape the Weight Space of Deep Neural Networks?cited by 4
  36. 2024Theoretical Proof that Auto-regressive Language Models Collapse when Real-world Data is a Finite Setcited by 4
  37. 2025Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Themcited by 4
  38. 2025Stabilizing Self-Consuming Diffusion Models with Latent Space Filteringcited by 4

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better
  2. 2023Improving the Effectiveness of Deep Generative Data
  3. 2024Mind the Gap Between Synthetic and Real: Utilizing Transfer Learning to Probe the Boundaries of Stable Diffusion Generated Data
  4. 2025Boosting Statistic Learning with Synthetic Data from Pretrained Large Models
  5. 2024Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Images
  6. 2026Memorisation, convergence and generalisation in generative models
  7. 2023Diversify, Don't Fine-Tune: Scaling Up Visual Recognition Training with Synthetic Images
  8. 2024Controlling the Fidelity and Diversity of Deep Generative Models via Pseudo Density