mlsanityData to 13 Oct 2026

Self-Consuming Generative Models Go MAD

arXiv 2307.01850 ↗First posted 4 Jul 2023cs.LG cs.AI cs.CVcited by 246

Not on the map. Why?

Cited by 73 in Synthetic data and self-improvement for LLMs, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Seismic advances in generative AI algorithms for imagery, text, and other data types has led to the temptation to use synthetic data to train next-generation models. Repeating this process creates an autophagous (self-consuming) loop whose properties are poorly understood. We conduct a thorough analytical and empirical analysis using state-of-the-art generative image models of three families of autophagous loops that differ in how fixed or fresh real training data is available through the generations of training and in whether the samples from previous generation models have been biased to trade off data quality versus diversity. Our primary conclusion across all scenarios is that without enough fresh real data in each generation of an autophagous loop, future generative models are doomed to have their quality (precision) or diversity (recall) progressively decrease. We term this condition Model Autophagy Disorder (MAD), making analogy to mad cow disease.

Key facts. Cited by 246 papers in the map (data through 13 Oct 2026). Builds on 58 of its 92 references that are in the map. First posted on arXiv 4 Jul 2023.

Builds on

58 of 92 references.

References · 58

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  3. 2023GPT-4 Technical Reportcited by 18,278
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  5. 2020Denoising Diffusion Implicit Modelscited by 9,718
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,564
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,000
  8. 2022Classifier-Free Diffusion Guidancecited by 5,785
  9. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,516
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,141
  12. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,585
38 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,019
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,619
  4. 2017Improved Training of Wasserstein GANscited by 2,226
  5. 2016Density estimation using Real NVPcited by 2,033
  6. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,542
  7. 2021Alias-Free Generative Adversarial Networkscited by 1,183
  8. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 928
  9. 2023A Watermark for Large Language Modelscited by 717
  10. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 717
  11. 2022Large Language Models Can Self-Improvecited by 659
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 613
  13. 2022DiGress: Discrete Denoising diffusion for graph generationcited by 509
  14. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 477
  15. 2023MusicLM: Generating Music From Textcited by 445
  16. 2022TabDDPM: Modelling Tabular Data with Diffusion Modelscited by 376
  17. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 349
  18. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 327
  19. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  20. 2023The Stable Signature: Rooting Watermarks in Latent Diffusion Modelscited by 303
  21. 2022Brain Imaging Generation with Latent Diffusion Modelscited by 267
  22. 2023Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robustcited by 187
  23. 2023On the Reliability of Watermarks for Large Language Modelscited by 187
  24. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 172
  25. 2023A Recipe for Watermarking Diffusion Modelscited by 141
  26. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 128
  27. 2023The Science of Detecting LLM-Generated Textscited by 128
  28. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 103
  29. 2022Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generationcited by 99
  30. 2017CAN: Creative Adversarial Networks, Generating "Art" by Learning About Styles and Deviating from Style Normscited by 96
  31. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  32. 2023ChatGPT is not all you need. A State of the Art Review of large Generative AI modelscited by 86
  33. 2020DeepFake Detection by Analyzing Convolutional Tracescited by 81
  34. 2023Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusioncited by 81
  35. 2020Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Samplingcited by 71
  36. 2022Will Large-scale Generative Models Corrupt Future Datasets?cited by 59
  37. 2023Towards Understanding the Interplay of Generative Artificial Intelligence and the Internetcited by 54
  38. 2021OpenFWI: Large-Scale Multi-Structural Benchmark Datasets for Seismic Full Waveform Inversioncited by 44

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 19; 2024: 66; 2025: 70; 2026: 91.

246 papers.

Sort:

By area:

  1. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 697
  2. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 260
  3. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 158
  4. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 148
  5. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 128
  6. 2024Iterated Denoising Energy Matching for Sampling from Boltzmann Densitiescited by 115
  7. 2023A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directionscited by 104
  8. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 99
  9. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  10. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 98
  11. 2024How to Train Data-Efficient LLMscited by 91
  12. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
38 more
  1. 2025Flow Matching Policy Gradientscited by 90
  2. 2024A Survey on Self-Evolution of Large Language Modelscited by 88
  3. 2024Understanding Hallucinations in Diffusion Models through Mode Interpolationcited by 88
  4. 2024Model Collapse Demystified: The Case of Regressioncited by 81
  5. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 78
  6. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 69
  7. 2024Strong Model Collapsecited by 64
  8. 2024Fairness Feedback Loops: Training on Synthetic Data Amplifies Biascited by 55
  9. 2025The Best Instruction-Tuning Data are Those That Fitcited by 50
  10. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 49
  11. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 45
  12. 2025Reasoning to Learn from Latent Thoughtscited by 43
  13. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  14. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 39
  15. 2023Neural Retrievers are Biased Towards LLM-Generated Contentcited by 38
  16. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 38
  17. 2024Self-Correcting Self-Consuming Loops for Generative Model Trainingcited by 37
  18. 2024DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarioscited by 37
  19. 2024Self-Improving Diffusion Models with Synthetic Datacited by 35
  20. 2024Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talkcited by 34
  21. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 33
  22. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 30
  23. 2024Towards Theoretical Understandings of Self-Consuming Generative Modelscited by 28
  24. 2024DataDream: Few-shot Guided Dataset Generationcited by 28
  25. 2023Nepotistically Trained Generative-AI Models Collapsecited by 27
  26. 2024Universality of the π2/6 Pathway in Avoiding Model Collapsecited by 25
  27. 2024How to Synthesize Text Data without Model Collapse?cited by 24
  28. 2024Rate of Model Collapse in Recursive Trainingcited by 24
  29. 2023Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflowcited by 22
  30. 2024Heat Death of Generative Models in Closed-Loop Learningcited by 22
  31. 2024Image Generation Diversity Issues and How to Tame Themcited by 22
  32. 2024Organic or Diffused: Can We Distinguish Human Art from AI-generated Images?cited by 21
  33. 2024Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modelingcited by 20
  34. 2024Towards Reliable Verification of Unauthorized Data Usage in Personalized Text-to-Image Diffusion Modelscited by 20
  35. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 18
  36. 2025Ambient Diffusion Omni: Training Good Models with Bad Datacited by 18
  37. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  38. 2025A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loopscited by 16

50 most cited of 246 All 246 (JSON)