# Losing dimensions: Geometric memorization in generative diffusion

arXiv: https://arxiv.org/abs/2410.08727 · HTML: https://mlsanity.com/paper/2410.08727/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 31 of its 55 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 31)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [Convergence of denoising diffusion models under the manifold hypothesis](https://mlsanity.com/paper/2208.05314/) (2022)
- [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016)
- [Understanding and Mitigating Copying in Diffusion Models](https://mlsanity.com/paper/2305.20086/) (2023)
- [Generalization in diffusion models arises from geometry-adaptive harmonic representations](https://mlsanity.com/paper/2310.02557/) (2023)
- [Dynamical Regimes of Diffusion Models](https://mlsanity.com/paper/2402.18491/) (2024)
- [Score-Based Generative Models Detect Manifolds](https://mlsanity.com/paper/2206.01018/) (2022)
- [Spontaneous Symmetry Breaking in Generative Diffusion Models](https://mlsanity.com/paper/2305.19693/) (2023)
- [Verifying the Union of Manifolds Hypothesis for Image Data](https://mlsanity.com/paper/2207.02862/) (2022)
- [A Reproducible Extraction of Training Images from Diffusion Models](https://mlsanity.com/paper/2305.08694/) (2023)
- [Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributions](https://mlsanity.com/paper/2409.02426/) (2024)
- [In search of dispersed memories: Generative diffusion models are associative memory networks](https://mlsanity.com/paper/2309.17290/) (2023)
- [A Geometric View of Data Complexity: Efficient Local Intrinsic Dimension Estimation with Diffusion Models](https://mlsanity.com/paper/2406.03537/) (2024)
- [Sampling, Diffusions, and Stochastic Localization](https://mlsanity.com/paper/2305.10690/) (2023)
- [The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability](https://mlsanity.com/paper/2310.17467/) (2023)
- [Your diffusion model secretly knows the dimension of the data manifold](https://mlsanity.com/paper/2212.12611/) (2022)
- [Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusion](https://mlsanity.com/paper/2410.05898/) (2024)
- [On gauge freedom, conservativity and intrinsic dimensionality estimation in diffusion models](https://mlsanity.com/paper/2402.03845/) (2024)

## Built on by (most cited first, 29 of 29)

- [Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training](https://mlsanity.com/paper/2505.17638/) (2025)
- [Memorization to Generalization: Emergence of Diffusion Models from Associative Memory](https://mlsanity.com/paper/2505.21777/) (2025)
- [Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models](https://mlsanity.com/paper/2505.16959/) (2025)
- [Denoising Score Matching with Random Features: Insights on Diffusion Models from Precise Learning Curves](https://mlsanity.com/paper/2502.00336/) (2025)
- [Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling](https://mlsanity.com/paper/2502.05743/) (2025)
- [On the Interpolation Effect of Score Smoothing in Diffusion Models](https://mlsanity.com/paper/2502.19499/) (2025)
- [TCFG: Tangential Damping Classifier-free Guidance](https://mlsanity.com/paper/2503.18137/) (2025)
- [A solvable model of learning generative diffusion: theory and insights](https://mlsanity.com/paper/2501.03937/) (2025)
- [Carr'e du champ flow matching: better quality-generalisation tradeoff in generative models](https://mlsanity.com/paper/2510.05930/) (2025)
- [The Information Dynamics of Generative Diffusion](https://mlsanity.com/paper/2508.19897/) (2025)
- [Analysis of Diffusion Models for Manifold Data](https://mlsanity.com/paper/2502.04339/) (2025)
- [Generalization of Diffusion Models Arises with a Balanced Representation Space](https://mlsanity.com/paper/2512.20963/) (2025)
- [Local Coverage Governs Memorization in Diffusion Models](https://mlsanity.com/paper/2606.14390/) (2026)
- [When and how can inexact generative models still sample from the data manifold?](https://mlsanity.com/paper/2508.07581/) (2025)
- [Emergence of Distortions in High-Dimensional Guided Diffusion Models](https://mlsanity.com/paper/2602.00716/) (2026)
- [Steering Dynamical Regimes of Diffusion Models by Breaking Detailed Balance](https://mlsanity.com/paper/2602.15914/) (2026)
- [Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data](https://mlsanity.com/paper/2603.22962/) (2026)
- [Generalization, memorization, and overfitting for diffusion models trained in the lazy high-dimensional regime](https://mlsanity.com/paper/2608.23938/) (2026)
- [Diffusion Models Memorize in Training -- and Generalize in Inference](https://mlsanity.com/paper/2603.13419/) (2026)
- [Manifold Generalization Provably Proceeds Memorization in Diffusion Models](https://mlsanity.com/paper/2603.23792/) (2026)
- [The Interplay of Data Structure and Imbalance in the Learning Dynamics of Diffusion Models](https://mlsanity.com/paper/2605.06367/) (2026)
- [Estimating Dataset Dimension via Singular Metrics under the Manifold Hypothesis: Application to Inverse Problems](https://mlsanity.com/paper/2507.07291/) (2025)
- [On the Limits of Latent Reuse in Diffusion Models](https://mlsanity.com/paper/2605.13448/) (2026)
- [Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles](https://mlsanity.com/paper/2606.09718/) (2026)
- [Double Descent and Malign Overfitting in Diffusion Models](https://mlsanity.com/paper/2609.26392/) (2026)
- [Latent Iterative Refinement Flow: A Geometric Constrained Approach for Few-Shot Generation](https://mlsanity.com/paper/2509.19903/) (2025)
- [Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data](https://mlsanity.com/paper/2604.26841/) (2026)
- [First Learn, Then Memorize: The Spectral Bias of Diffusion Models](https://mlsanity.com/paper/2609.35377/) (2026)
- [Loss Curvature as a Sampling-Free Proxy for Memorization in Diffusion Models](https://mlsanity.com/paper/2610.19042/) (2026)

## Abstract (arXiv)

Diffusion models power leading generative AI, but when and how they memorize training data, especially on low-dimensional manifolds, remains unclear. We find memorization emerges gradually, not abruptly: as data become scarce, diffusion models experience a smooth collapse where their capacity to vary across independent directions diminishes. Measuring latent dimensionality via the learned score field, we reveal how generative behavior increasingly centers on a few examples while other variations "freeze out". We propose a geometric memorization theory, showing that salient features collapse first, then finer details, leading to near point-wise replication. This mirrors physical systems condensing into a few low-energy configurations. Our theoretical predictions align with both synthetic and real data, identifying geometric memorization as a distinct phase between generalization and exact copying.