# How Diffusion Models Memorize

arXiv: https://arxiv.org/abs/2509.25705 · HTML: https://mlsanity.com/paper/2509.25705/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 27 of its 31 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [Understanding and Mitigating Copying in Diffusion Models](https://mlsanity.com/paper/2305.20086/) (2023)
- [Generalization in diffusion models arises from geometry-adaptive harmonic representations](https://mlsanity.com/paper/2310.02557/) (2023)
- [A Self-Supervised Descriptor for Image Copy Detection](https://mlsanity.com/paper/2202.10261/) (2022)
- [The Privacy Onion Effect: Memorization is Relative](https://mlsanity.com/paper/2206.10469/) (2022)
- [Ambient Diffusion: Learning Clean Distributions from Corrupted Data](https://mlsanity.com/paper/2305.19256/) (2023)
- [Detecting, Explaining, and Mitigating Memorization in Diffusion Models](https://mlsanity.com/paper/2407.21720/) (2024)
- [A Reproducible Extraction of Training Images from Diffusion Models](https://mlsanity.com/paper/2305.08694/) (2023)
- [On Memorization in Probabilistic Deep Generative Models](https://mlsanity.com/paper/2106.03216/) (2021)
- [Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attention](https://mlsanity.com/paper/2403.11052/) (2024)
- [On the De-duplication of LAION-2B](https://mlsanity.com/paper/2303.12733/) (2023)
- [A Geometric Framework for Understanding Memorization in Generative Models](https://mlsanity.com/paper/2411.00113/) (2024)
- [Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Models](https://mlsanity.com/paper/2406.02366/) (2024)
- [Classifier-Free Guidance inside the Attraction Basin May Cause Memorization](https://mlsanity.com/paper/2411.16738/) (2024)
- [Understanding and Mitigating Memorization in Generative Models via Sharpness of Probability Landscapes](https://mlsanity.com/paper/2412.04140/) (2024)

## Built on by (most cited first, 6 of 6)

- [Dominant vs. Dominated: Concept-Level Generative Collapse in Diffusion Models](https://mlsanity.com/paper/2512.20666/) (2025)
- [Memorization Control in Diffusion Models from Denoising-centric Perspective](https://mlsanity.com/paper/2601.21348/) (2026)
- [Steering Away from Memorization: Reachability-Constrained Reinforcement Learning for Text-to-Image Diffusion](https://mlsanity.com/paper/2603.00140/) (2026)
- [The Persistence of Cultural Memory: Investigating Multimodal Iconicity in Diffusion Models](https://mlsanity.com/paper/2511.11435/) (2025)
- [Memorization In Stable Diffusion Is Unexpectedly Driven by CLIP Embeddings](https://mlsanity.com/paper/2605.02908/) (2026)
- [Support-Conditioned Flow Matching Is Kernel Smoothing](https://mlsanity.com/paper/2605.13386/) (2026)

## Abstract (arXiv)

Despite their success in image generation, diffusion models can memorize training data, raising serious privacy and copyright concerns. Although prior work has sought to characterize, detect, and mitigate memorization, the fundamental question of why and how it occurs remains unresolved. In this paper, we revisit the diffusion and denoising process and analyze latent space dynamics to address the question: "How do diffusion models memorize?" We show that memorization is driven by the overestimation of training samples during early denoising, which reduces diversity, collapses denoising trajectories, and accelerates convergence toward the memorized image. Specifically: (i) memorization cannot be explained by overfitting alone, as training loss is larger under memorization due to classifier-free guidance amplifying predictions and inducing overestimation; (ii) memorized prompts inject training images into noise predictions, forcing latent trajectories to converge and steering denoising toward their paired samples; and (iii) a decomposition of intermediate latents reveals how initial randomness is quickly suppressed and replaced by memorized content, with deviations from the theoretical denoising schedule correlating almost perfectly with memorization severity. Together, these results identify early overestimation as the central underlying mechanism of memorization in diffusion models.