mlsanityData to 14 Oct 2026

A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective

arXiv 2509.16499 ↗First posted 20 Sep 2025cs.LGcited by 18

On the map: Diffusion models → Diffusion foundations and sampling

also in 1 more
Abstract (arXiv)

The widespread use of diffusion models has led to an abundance of AI-generated data, raising concerns about model collapse -- a phenomenon in which recursive iterations of training on synthetic data lead to performance degradation. Prior work primarily characterizes this collapse via variance shrinkage or distribution shift, but these perspectives miss practical manifestations of model collapse. This paper identifies a transition from generalization to memorization during model collapse in diffusion models, where models increasingly replicate training data instead of generating novel content during iterative training on synthetic samples. This transition is directly driven by the declining entropy of the synthetic training data produced in each training cycle, which serves as a clear indicator of model degradation. Motivated by this insight, we propose an entropy-based data selection strategy to mitigate the transition from generalization to memorization and alleviate model collapse. Empirical results show that our approach significantly enhances visual quality and diversity in recursive generation, effectively preventing collapse.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 54 of its 76 references that are in the map. First posted on arXiv 20 Sep 2025.

Builds on

54 of 76 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Towards Understanding the Mechanisms of Classifier-Free Guidance25 May
  2. 2025Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling9 Feb
  3. 2024The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications12 Dec
  4. 2024Analyzing and Mitigating Model Collapse in Rectified Flow Models11 Dec
  5. 2024Towards a Mechanistic Explanation of Diffusion Model Generalization28 Nov
  6. 2024Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure31 Oct
  7. 2024On Inductive Biases That Enable Generalization of Diffusion Transformers28 Oct
  8. 2024What does guidance do? A fine-grained analysis in a simple setting19 Sep
  9. 2024Exploring Low-Dimensional Subspaces in Diffusion Models for Controllable Image Editing4 Sep
  10. 2024Self-Improving Diffusion Models with Synthetic Data29 Aug
  11. 2024Detecting, Explaining, and Mitigating Memorization in Diffusion Models31 Jul
  12. 2024Evaluating the design space of diffusion-based generative models18 Jun
5 more
  1. 2023On Memorization in Diffusion Models4 Oct
  2. 2023Generalization in diffusion models arises from geometry-adaptive harmonic representations4 Oct
  3. 2023Understanding and Mitigating Copying in Diffusion Models31 May
  4. 2022Classifier-Free Diffusion Guidance26 Jul
  5. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 37

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  6. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  7. 2023Extracting Training Data from Diffusion Modelscited by 732
  8. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  9. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 424
  10. 2023Self-Consuming Generative Models Go MADcited by 250
  11. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 214
  12. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 205
25 more
  1. 2021How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Modelscited by 196
  2. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 151
  3. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 130
  4. 2024An analytic theory of creativity in convolutional diffusion modelscited by 104
  5. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 100
  6. 2024Model Collapse Demystified: The Case of Regressioncited by 82
  7. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapsecited by 79
  8. 2024Strong Model Collapsecited by 65
  9. 2024Fairness Feedback Loops: Training on Synthetic Data Amplifies Biascited by 55
  10. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 52
  11. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 45
  12. 2024A Geometric Framework for Understanding Memorization in Generative Modelscited by 40
  13. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 39
  14. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 33
  15. 2024Towards Theoretical Understandings of Self-Consuming Generative Modelscited by 29
  16. 2023Towards a statistical theory of data selection under weak supervisioncited by 27
  17. 2023Nepotistically Trained Generative-AI Models Collapsecited by 27
  18. 2024Universality of the π2/6 Pathway in Avoiding Model Collapsecited by 26
  19. 2024How to Synthesize Text Data without Model Collapse?cited by 25
  20. 2024Rate of Model Collapse in Recursive Trainingcited by 25
  21. 2025Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spacescited by 21
  22. 2025Improving the Scaling Laws of Synthetic Data with Deliberate Practicecited by 17
  23. 2025A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loopscited by 16
  24. 2024Model Collapse in the Self-Consuming Chain of Diffusion Finetuning: A Novel Perspective from Quantitative Trait Modelingcited by 12
  25. 2025Generating on Generated: An Approach Towards Self-Evolving Diffusion Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 14.

18 papers.

Sort:

By area:

  1. 2025Generalization of Diffusion Models Arises with a Balanced Representation Spacecited by 11
  2. 2025From Navigation to Refinement: Revealing the Two-Stage Nature of Flow-based Diffusion Models through Oracle Velocitycited by 8
  3. 2026Quantifying Error Propagation and Model Collapse in Diffusion Modelscited by 4
  4. 2026Ambient Dataloops: Generative Models for Dataset Refinementcited by 2
  5. 2026Manifold Generalization Provably Proceeds Memorization in Diffusion Modelscited by 2
  6. 2025Rethinking Test Time Scaling for Flow-Matching Generative Modelscited by 1
  7. 2026Diffusion Models Preferentially Memorize Prototypical Examples or: Why Does My Diffusion Model Love Slop?cited by 1
  8. 2026Evaluating the Representation Space of Diffusion Models via Self-Supervised Principlescited by 1
  9. 2026When Sample Selection Bias Precipitates Model Collapsecited by 1
  10. 2025Coarse-to-Fine Hierarchical Alignment for UAV-based Human Detection using Diffusion Modelscited by 0
  11. 2026A Markovian View of Iterative-Feedback Loops in Image Generative Models: Neural Resonance and Model Collapsecited by 0
  12. 2026Entropy Minimization without Model Collapse: Mitigating Prediction Bias in Medical Imagingcited by 0
6 more
  1. 2026A Theoretical Analysis of Memory and Overfitting Phenomena in Stochastic Interpolation Modelscited by 0
  2. 2026Recursively Trained Diffusion Models: Limiting Collapse Distribution and Spectral Characterizationcited by 0
  3. 2026Minimax bounds for watermarked and masked recursive discrete distribution estimationcited by 0
  4. 2026Misalignment of Low-Loss Regions Causes Grokkingcited by 0
  5. 2026Scaling and Distilling Text Embeddings for Better Diffusibilitycited by 0
  6. 2026Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Edge of Memorization in Diffusion Models
  2. 2025Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models
  3. 2025Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training
  4. 2026Understanding diffusion models requires rethinking (again) generalization
  5. 2025ForTIFAI: Fending Off Recursive Training Induced Failure for AI Model Collapse
  6. 2023The Emergence of Reproducibility and Generalizability in Diffusion Models
  7. 2026Memorisation, convergence and generalisation in generative models
  8. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence