# Towards Theoretical Understandings of Self-Consuming Generative Models

arXiv: https://arxiv.org/abs/2402.11778 · HTML: https://mlsanity.com/paper/2402.11778/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 24 of 24)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Can AI-Generated Text be Reliably Detected?](https://mlsanity.com/paper/2303.11156/) (2023)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [Data Augmentation Generative Adversarial Networks](https://mlsanity.com/paper/1711.04340/) (2017)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [On the Stability of Iterative Retraining of Generative Models on their own Data](https://mlsanity.com/paper/2310.00429/) (2023)
- [Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loop](https://mlsanity.com/paper/2311.16822/) (2023)
- [On the Generalization Properties of Diffusion Models](https://mlsanity.com/paper/2311.01797/) (2023)
- [Towards Understanding the Interplay of Generative Artificial Intelligence and the Internet](https://mlsanity.com/paper/2306.06130/) (2023)
- [The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descent](https://mlsanity.com/paper/2305.17490/) (2023)
- [Combining Generative Artificial Intelligence (AI) and the Internet: Heading towards Evolution or Degradation?](https://mlsanity.com/paper/2303.01255/) (2023)
- [Overcoming Barriers to Data Sharing with Medical Image Generation: A Comprehensive Evaluation](https://mlsanity.com/paper/2012.03769/) (2020)
- [Concentration in unbounded metric spaces and algorithmic stability](https://mlsanity.com/paper/1309.1007/) (2013)
- [A Priori Estimates of the Population Risk for Residual Networks](https://mlsanity.com/paper/1903.02154/) (2019)
- [Generalization and Memorization: The Bias Potential Model](https://mlsanity.com/paper/2011.14269/) (2020)
- [Do You Trust ChatGPT? -- Perceived Credibility of Human and AI-Generated Content](https://mlsanity.com/paper/2309.02524/) (2023)
- [A Mathematical Framework for Learning Probability Distributions](https://mlsanity.com/paper/2212.11481/) (2022)
- [Radon Sobolev Variational Auto-Encoders](https://mlsanity.com/paper/1911.13135/) (2019)

## Built on by (most cited first, 29 of 29)

- [A Survey on Self-Evolution of Large Language Models](https://mlsanity.com/paper/2404.14387/) (2024)
- [Understanding Hallucinations in Diffusion Models through Mode Interpolation](https://mlsanity.com/paper/2406.09358/) (2024)
- [How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse](https://mlsanity.com/paper/2404.05090/) (2024)
- [A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective](https://mlsanity.com/paper/2509.16499/) (2025)
- [A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops](https://mlsanity.com/paper/2502.18865/) (2025)
- [When Models Don't Collapse: On the Consistency of Iterative MLE](https://mlsanity.com/paper/2505.19046/) (2025)
- [Model Collapse in the Self-Consuming Chain of Diffusion Finetuning: A Novel Perspective from Quantitative Trait Modeling](https://mlsanity.com/paper/2407.17493/) (2024)
- [LLM as a Broken Telephone: Iterative Generation Distorts Information](https://mlsanity.com/paper/2502.20258/) (2025)
- [Analyzing and Mitigating Model Collapse in Rectified Flow Models](https://mlsanity.com/paper/2412.08175/) (2024)
- [Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gap](https://mlsanity.com/paper/2507.00075/) (2025)
- [Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence](https://mlsanity.com/paper/2510.16657/) (2025)
- [Language Generation with Replay: A Learning-Theoretic View of Model Collapse](https://mlsanity.com/paper/2603.11784/) (2026)
- [Stabilizing Self-Consuming Diffusion Models with Latent Space Filtering](https://mlsanity.com/paper/2511.12742/) (2025)
- [Quantifying Error Propagation and Model Collapse in Diffusion Models](https://mlsanity.com/paper/2602.16601/) (2026)
- [Review of Hallucination Understanding in Large Language and Vision Models](https://mlsanity.com/paper/2510.00034/) (2025)
- [When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loop](https://mlsanity.com/paper/2605.29267/) (2026)
- [Shallow Diffuse: Robust and Invisible Watermarking through Low-Dimensional Subspaces in Diffusion Models](https://mlsanity.com/paper/2410.21088/) (2024)
- [HRP: High-Rank Preheating for Superior LoRA Initialization](https://mlsanity.com/paper/2502.07739/) (2025)
- [Large Language Models are Shannon Lossy Compressors Not Solomonoff Induction Estimators: Self-improvement and Singularity Are Not Near Without Symbolic Model Synthesis](https://mlsanity.com/paper/2601.05280/) (2026)
- [From Collapse to Improvement: Statistical Perspectives on the Evolutionary Dynamics of Iterative Training on Contaminated Sources](https://mlsanity.com/paper/2602.10531/) (2026)
- [ALIGN-FL: Architecture-independent Learning through Invariant Generative component sharing in Federated Learning](https://mlsanity.com/paper/2512.13316/) (2025)
- [A Task-Centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula](https://mlsanity.com/paper/2602.10014/) (2026)
- [A Markovian View of Iterative-Feedback Loops in Image Generative Models: Neural Resonance and Model Collapse](https://mlsanity.com/paper/2602.19033/) (2026)
- [Recursively Trained Diffusion Models: Limiting Collapse Distribution and Spectral Characterization](https://mlsanity.com/paper/2606.13796/) (2026)
- [Optimal Self-Distillation for Rectified Flow via Linear Probing](https://mlsanity.com/paper/2607.14947/) (2026)
- [Minimax bounds for watermarked and masked recursive discrete distribution estimation](https://mlsanity.com/paper/2608.31091/) (2026)
- [SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Data](https://mlsanity.com/paper/2609.12353/) (2026)
- [Recursive LLM Degradation in Biomedical Question Answering: A Cross-Generation Study](https://mlsanity.com/paper/2609.34257/) (2026)
- [Screening Collapse under Recursive Synthetic Data](https://mlsanity.com/paper/2610.15617/) (2026)

## Abstract (arXiv)

This paper tackles the emerging challenge of training generative models within a self-consuming loop, wherein successive generations of models are recursively trained on mixtures of real and synthetic data from previous generations. We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models, including parametric and non-parametric models. Specifically, we derive bounds on the total variation (TV) distance between the synthetic data distributions produced by future models and the original real data distribution under various mixed training scenarios for diffusion models with a one-hidden-layer neural network score function. Our analysis demonstrates that this distance can be effectively controlled under the condition that mixed training dataset sizes or proportions of real data are large enough. Interestingly, we further unveil a phase transition induced by expanding synthetic data amounts, proving theoretically that while the TV distance exhibits an initial ascent, it declines beyond a threshold point. Finally, we present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.