mlsanityData to 7 Oct 2026

On Memorization in Diffusion Models

arXiv 2310.02664 ↗First posted 4 Oct 2023cs.LG cs.AI cs.CVcited by 122

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Due to their capacity to generate novel and high-quality samples, diffusion models have attracted significant research interest in recent years. Notably, the typical training objective of diffusion models, i.e., denoising score matching, has a closed-form optimal solution that can only generate training data replicating samples. This indicates that a memorization behavior is theoretically expected, which contradicts the common generalization ability of state-of-the-art diffusion models, and thus calls for a deeper understanding. Looking into this, we first observe that memorization behaviors tend to occur on smaller-sized datasets, which motivates our definition of effective model memorization (EMM), a metric measuring the maximum size of training data at which a learned diffusion model approximates its theoretical optimum. Then, we quantify the impact of the influential factors on these memorization behaviors in terms of EMM, focusing primarily on data distribution, model configuration, and training procedure. Besides comprehensive empirical results identifying the influential factors, we surprisingly find that conditioning training data on uninformative random labels can significantly trigger the memorization in diffusion models. Our study holds practical significance for diffusion model users and offers clues to theoretical research in deep generative models. Code is available at https://github.com/sail-sg/DiffMemorize.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 46 of its 64 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

46 of 64 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Understanding and Mitigating Copying in Diffusion Models31 May
  2. 2023On the Generalization of Diffusion Model24 May
  3. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  5. 2021Variational Diffusion Models1 Jul
  6. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  7. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  9. 2020Denoising Diffusion Implicit Models6 Oct
  10. 2020Denoising Diffusion Probabilistic Models19 Jun
  11. 2020Improved Techniques for Training Score-Based Generative Models16 Jun

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  12. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
23 more
  1. 2020Extracting Training Data from Large Language Modelscited by 2,144
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  5. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  6. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  7. 2023Extracting Training Data from Diffusion Modelscited by 723
  8. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  9. 2019Benign Overfitting in Linear Regressioncited by 544
  10. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 511
  11. 2022DiGress: Discrete Denoising diffusion for graph generationcited by 503
  12. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  13. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  14. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
  15. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  16. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  17. 2023A Recipe for Watermarking Diffusion Modelscited by 136
  18. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidancecited by 84
  19. 2021On the geometry of generalization and memorization in deep neural networkscited by 71
  20. 2023Can Neural Network Memorization Be Localized?cited by 66
  21. 2021On Memorization in Probabilistic Deep Generative Modelscited by 60
  22. 2022When do GANs replicate? On the choice of dataset sizecited by 34
  23. 2021This Person (Probably) Exists. Identity Membership Attacks Against GAN Generated Facescited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 33; 2025: 53; 2026: 34.

122 papers.

Sort:

By area:

  1. 2024An analytic theory of creativity in convolutional diffusion modelscited by 103
  2. 2025Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Trainingcited by 78
  3. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  4. 2024Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributionscited by 53
  5. 2025On the Closed-Form of Flow Matching: Generalization Does Not Arise from Target Stochasticitycited by 48
  6. 2023Intriguing Properties of Data Attribution on Diffusion Modelscited by 47
  7. 2024Towards a Mechanistic Explanation of Diffusion Model Generalizationcited by 42
  8. 2024A Geometric Framework for Understanding Memorization in Generative Modelscited by 40
  9. 2025Memorization and Regularization in Generative Diffusion Modelscited by 39
  10. 2024Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structurecited by 38
  11. 2024A Good Score Does not Lead to A Good Generative Modelcited by 36
  12. 2024Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Modelscited by 36
38 more
  1. 2025Memorization to Generalization: Emergence of Diffusion Models from Associative Memorycited by 35
  2. 2024The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright Breaches Without Adjusting Finetuning Pipelinecited by 33
  3. 2024Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defensescited by 30
  4. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  5. 2025Locality in Image Diffusion Models Emerges from Data Statisticscited by 25
  6. 2025Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Modelscited by 24
  7. 2025On the Edge of Memorization in Diffusion Modelscited by 22
  8. 2023The Emergence of Reproducibility and Generalizability in Diffusion Modelscited by 21
  9. 2024Understanding and Mitigating Memorization in Generative Models via Sharpness of Probability Landscapescited by 20
  10. 2025Denoising Score Matching with Random Features: Insights on Diffusion Models from Precise Learning Curvescited by 20
  11. 2025Diffusion Models and the Manifold Hypothesis: Log-Domain Smoothing is Geometry Adaptivecited by 20
  12. 2024Hidden in the Noise: Two-Stage Robust Watermarking for Imagescited by 18
  13. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  14. 2025Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusioncited by 17
  15. 2025Selective Underfitting in Diffusion Modelscited by 17
  16. 2025Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorizationcited by 16
  17. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  18. 2025On the Interpolation Effect of Score Smoothing in Diffusion Modelscited by 15
  19. 2024Wasserstein proximal operators describe score-based generative models and resolve memorizationcited by 14
  20. 2024Deep Generative Models for 3D Medical Image Synthesiscited by 14
  21. 2025Provable Separations between Memorization and Generalization in Diffusion Modelscited by 14
  22. 2024Evaluating and Mitigating IP Infringement in Visual Generative AIcited by 13
  23. 2024Memorized Images in Diffusion Models share a Subspace that can be Located and Deletedcited by 13
  24. 2024Spatially-Aware Diffusion Models with Cross-Attention for Global Field Reconstruction with Sparse Observationscited by 13
  25. 2025Carr'e du champ flow matching: better quality-generalisation tradeoff in generative modelscited by 13
  26. 2024Iterative Ensemble Training with Anti-Gradient Control for Mitigating Memorization in Diffusion Modelscited by 12
  27. 2025Generative Modeling of Weights: Generalization or Memorization?cited by 12
  28. 2024Replication in Visual Diffusion Models: A Survey and Outlookcited by 11
  29. 2024Slight Corruption in Pre-training Data Makes Better Diffusion Modelscited by 9
  30. 2025Demystifying Diffusion Policies: Action Memorization and Simple Lookup Table Alternativescited by 9
  31. 2024Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Expertscited by 8
  32. 2024U Can't Gen This? A Survey of Intellectual Property Protection Methods for Data in Generative AIcited by 7
  33. 2024SIDE: Surrogate Conditional Data Extraction from Diffusion Modelscited by 7
  34. 2024Understanding and Mitigating Memorization in Diffusion Models for Tabular Datacited by 7
  35. 2025Efficient Flow Matching using Latent Variablescited by 6
  36. 2025The Generation Phases of Flow Matching: a Denoising Perspectivecited by 6
  37. 2024MemBench: Memorized Image Trigger Prompt Dataset for Diffusion Modelscited by 5
  38. 2024Investigating Memorization in Video Diffusion Modelscited by 5

50 most cited of 122 All 122 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025How Diffusion Models Memorize
  2. 2026Local Coverage Governs Memorization in Diffusion Models
  3. 2024An Inversion-based Measure of Memorization for Diffusion Models
  4. 2024Detecting, Explaining, and Mitigating Memorization in Diffusion Models
  5. 2025Generalization of Diffusion Models Arises with a Balanced Representation Space
  6. 2025Quantifying the Ease of Reproducing Training Data in Unconditional Diffusion Models
  7. 2026First Learn, Then Memorize: The Spectral Bias of Diffusion Models
  8. 2026Diffusion Models Memorize in Training -- and Generalize in Inference