mlsanityData to 7 Oct 2026

On Memorization in Probabilistic Deep Generative Models

arXiv 2106.03216 ↗First posted 6 Jun 2021cs.LG stat.MLcited by 60

Not on the map. Why?

Cited by 20 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in deep generative models have led to impressive results in a variety of application domains. Motivated by the possibility that deep learning models might memorize part of the input data, there have been increased efforts to understand how memorization arises. In this work, we extend a recently proposed measure of memorization for supervised learning (Feldman, 2019) to the unsupervised density estimation problem and adapt it to be more computationally efficient. Next, we present a study that demonstrates how memorization can occur in probabilistic deep generative models such as variational autoencoders. This reveals that the form of memorization to which these models are susceptible differs fundamentally from mode collapse and overfitting. Furthermore, we show that the proposed memorization score measures a phenomenon that is not captured by commonly-used nearest neighbor tests. Finally, we discuss several strategies that can be used to limit memorization in practice. Our work thus provides a framework for understanding problematic memorization in probabilistic generative models.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 46 of its 80 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

46 of 80 references.

References · 46

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2013Auto-Encoding Variational Bayescited by 3,980
  8. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  9. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  10. 2016Improved Techniques for Training GANscited by 2,938
  11. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  12. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
34 more
  1. 2015Variational Inference with Normalizing Flowscited by 2,161
  2. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2017A Closer Look at Memorization in Deep Networkscited by 936
  6. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  7. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  8. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  9. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  10. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  11. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  12. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  13. 2015Importance Weighted Autoencoderscited by 454
  14. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  15. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  16. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 414
  17. 2017Deep Models Under the GAN: Information Leakage from Collaborative Deep Learningcited by 326
  18. 2015A note on the evaluation of generative modelscited by 291
  19. 2016Revisiting Classifier Two-Sample Testscited by 258
  20. 2020Influence Functions in Deep Learning Are Fragilecited by 247
  21. 2020Characterizing Structural Regularities of Labeled Data in Overparameterized Modelscited by 93
  22. 2018An empirical study on evaluation metrics of generative adversarial networkscited by 81
  23. 2021On the geometry of generalization and memorization in deep neural networkscited by 71
  24. 2013RNADE: The real-valued neural autoregressive density-estimatorcited by 65
  25. 2019Do We Train on Test Data? Purging CIFAR of Near-Duplicatescited by 61
  26. 2020A Non-Parametric Test to Detect Data-Copying in Generative Modelscited by 52
  27. 2019Detecting Overfitting of Deep Generative Networks via Latent Recoverycited by 44
  28. 2016On the Quantitative Analysis of Decoder-Based Generative Modelscited by 40
  29. 2014Differential Privacy: An Economic Method for Choosing Epsiloncited by 34
  30. 2020Towards GAN Benchmarks Which Require Generalizationcited by 31
  31. 2021Understanding Instance-based Interpretability of Variational Auto-Encoderscited by 21
  32. 2019Robust Variational Autoencoders for Outlier Detection and Repair of Mixed-Type Datacited by 16
  33. 2021Influence Estimation for Generative Adversarial Networkscited by 11
  34. 2019Robust Variational Autoencodercited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 19; 2024: 12; 2025: 14; 2026: 9.

60 papers.

Sort:

By area:

  1. 2023Extracting Training Data from Diffusion Modelscited by 723
  2. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  3. 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 237
  4. 2022Training Data Influence Analysis and Estimation: A Surveycited by 134
  5. 2023On Memorization in Diffusion Modelscited by 122
  6. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibrationcited by 90
  7. 2023Structure-informed Language Models Are Protein Designerscited by 68
  8. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  9. 2023The Journey, Not the Destination: How Data Guides Diffusion Modelscited by 42
  10. 2024Copyright Protection in Generative AI: A Technical Perspectivecited by 35
  11. 2024On the Challenges and Opportunities in Generative AIcited by 31
  12. 2023Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samplescited by 25
38 more
  1. 2023Federated Generative Learning with Foundation Modelscited by 23
  2. 2023Graph-level Representation Learning with Joint-Embedding Predictive Architecturescited by 21
  3. 2023A Probabilistic Fluctuation based Membership Inference Attack for Diffusion Modelscited by 17
  4. 2024Deep Generative Models for 3D Medical Image Synthesiscited by 14
  5. 2024Data Attribution for Diffusion Models: Timestep-induced Bias in Influence Estimationcited by 13
  6. 2022Data Isotopes for Data Provenance in DNNscited by 11
  7. 2023Gradient Surgery for One-shot Unlearning on Generative Modelcited by 11
  8. 2025Survey on the Evaluation of Generative Models in Musiccited by 10
  9. 2025Better Training Data Attribution via Better Inverse Hessian-Vector Productscited by 10
  10. 2025Unveiling Over-Memorization in Finetuning LLMs for Reasoning Taskscited by 10
  11. 2023Data-Copying in Generative Models: A Formal Frameworkcited by 9
  12. 2024ROME: Memorization Insights from Text, Logits and Representationcited by 8
  13. 2022A Pareto-optimal compositional energy-based model for sampling and optimization of protein sequencescited by 7
  14. 2024U Can't Gen This? A Survey of Intellectual Property Protection Methods for Data in Generative AIcited by 7
  15. 2024Understanding and Mitigating Memorization in Diffusion Models for Tabular Datacited by 7
  16. 2023SoK: Unintended Interactions among Machine Learning Defenses and Riskscited by 6
  17. 2024An Inversion-based Measure of Memorization for Diffusion Modelscited by 6
  18. 2025How Diffusion Models Memorizecited by 6
  19. 2023Quantifying Sample Anonymity in Score-Based Generative Models with Adversarial Fingerprintingcited by 4
  20. 2023Balancing Transparency and Risk: The Security and Privacy Risks of Open-Source Machine Learning Modelscited by 4
  21. 2024CollaFuse: Collaborative Diffusion Modelscited by 4
  22. 2025Machine Learners Should Acknowledge the Legal Implications of Large Language Models as Personal Datacited by 4
  23. 2023Structured Generative Models for Scene Understandingcited by 3
  24. 2023When Synthetic Data Met Regulationcited by 3
  25. 2025Kernel-Smoothed Scores for Denoising Diffusion: A Bias-Variance Studycited by 3
  26. 2026Biased Generalization in Diffusion Modelscited by 3
  27. 2023Preserving Privacy in GANs Against Membership Inference Attackcited by 2
  28. 2025Conditional Diffusion-Flow models for generating 3D cosmic density fields: applications to f(R) cosmologiescited by 2
  29. 2025Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysiscited by 2
  30. 2026Inference Attacks Against Graph Generative Diffusion Modelscited by 2
  31. 2026Diffusion Models Memorize in Training -- and Generalize in Inferencecited by 2
  32. 2022Unintended memorisation of unique features in neural networkscited by 1
  33. 2023SoK: Memorisation in machine learningcited by 1
  34. 2025A Closer Look on Memorization in Tabular Diffusion Model: A Data-Centric Perspectivecited by 1
  35. 2025Rethinking Memorization Measures and their Implications in Large Language Modelscited by 1
  36. 2025GriDiT: Factorized Grid-Based Diffusion for Efficient Long Image Sequence Generationcited by 1
  37. 2026Rethinking Anonymity Claims in Synthetic Data Generation: A Model-Centric Privacy Attack Perspectivecited by 1
  38. 2026Memorisation, convergence and generalisation in generative modelscited by 1

50 most cited of 60 All 60 (JSON)