# On the Generalization of Diffusion Model

arXiv: https://arxiv.org/abs/2305.14712 · HTML: https://mlsanity.com/paper/2305.14712/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 24 May 2023.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 21 of 21)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [Information-theoretic analysis of generalization capability of learning algorithms](https://mlsanity.com/paper/1705.07809/) (2017)
- [Generalization and Equilibrium in Generative Adversarial Nets (GANs)](https://mlsanity.com/paper/1703.00573/) (2017)
- [Tightening Mutual Information Based Bounds on Generalization Error](https://mlsanity.com/paper/1901.04609/) (2019)
- [Tighter expected generalization error bounds via Wasserstein distance](https://mlsanity.com/paper/2101.09315/) (2021)
- [Breaking Correlation Shift via Conditional Invariant Regularizer](https://mlsanity.com/paper/2207.06687/) (2022)
- [Characterization of Excess Risk for Locally Strongly Convex Population Risk](https://mlsanity.com/paper/2012.02456/) (2020)

## Built on by (most cited first, 30 of 30)

- [On Memorization in Diffusion Models](https://mlsanity.com/paper/2310.02664/) (2023)
- [On the Design Fundamentals of Diffusion Models: A Survey](https://mlsanity.com/paper/2306.04542/) (2023)
- [Closed-Form Diffusion Models](https://mlsanity.com/paper/2310.12395/) (2023)
- [Towards a Mechanistic Explanation of Diffusion Model Generalization](https://mlsanity.com/paper/2411.19339/) (2024)
- [A Geometric Framework for Understanding Memorization in Generative Models](https://mlsanity.com/paper/2411.00113/) (2024)
- [The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications](https://mlsanity.com/paper/2412.09726/) (2024)
- [Locality in Image Diffusion Models Emerges from Data Statistics](https://mlsanity.com/paper/2509.09672/) (2025)
- [The Emergence of Reproducibility and Generalizability in Diffusion Models](https://mlsanity.com/paper/2310.05264/) (2023)
- [Selective Underfitting in Diffusion Models](https://mlsanity.com/paper/2510.01378/) (2025)
- [Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization](https://mlsanity.com/paper/2502.03435/) (2025)
- [On the Interpolation Effect of Score Smoothing in Diffusion Models](https://mlsanity.com/paper/2502.19499/) (2025)
- [Flow Map Distillation Without Data](https://mlsanity.com/paper/2511.19428/) (2025)
- [Replication in Visual Diffusion Models: A Survey and Outlook](https://mlsanity.com/paper/2408.00001/) (2024)
- [Nearest Neighbour Score Estimators for Diffusion Generative Models](https://mlsanity.com/paper/2402.08018/) (2024)
- [Analyzing Neural Network-Based Generative Diffusion Models through Convex Optimization](https://mlsanity.com/paper/2402.01965/) (2024)
- [Algorithm- and Data-Dependent Generalization Bounds for Diffusion Models](https://mlsanity.com/paper/2506.03849/) (2025)
- [ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment](https://mlsanity.com/paper/2601.21484/) (2026)
- [Geometric Regularity in Deterministic Sampling Dynamics of Diffusion-based Generative Models](https://mlsanity.com/paper/2506.10177/) (2025)
- [FloodLense: A Framework for ChatGPT-based Real-time Flood Detection](https://mlsanity.com/paper/2401.15501/) (2024)
- [Improved Diffusion-based Generative Model with Better Adversarial Robustness](https://mlsanity.com/paper/2502.17099/) (2025)
- [Kernel-Smoothed Scores for Denoising Diffusion: A Bias-Variance Study](https://mlsanity.com/paper/2505.22841/) (2025)
- [An exact information theory of generalization phase transitions in Bayesian diffusion models](https://mlsanity.com/paper/2607.08041/) (2026)
- [Generative diffusion models from a PDE perspective](https://mlsanity.com/paper/2501.17054/) (2025)
- [A PDE Perspective on Generative Diffusion Models](https://mlsanity.com/paper/2511.05940/) (2025)
- [Enhancing Text-to-Image Editing via Hybrid Mask-Informed Fusion](https://mlsanity.com/paper/2405.15313/) (2024)
- [Generation Properties of Stochastic Interpolation under Finite Training Set](https://mlsanity.com/paper/2509.21925/) (2025)
- [Fragile Reconstruction: Adversarial Vulnerability of Reconstruction-Based Detectors for Diffusion-Generated Images](https://mlsanity.com/paper/2604.12781/) (2026)
- [OFA-Diffusion Compression: Compressing Diffusion Model in One-Shot Manner](https://mlsanity.com/paper/2604.12668/) (2026)
- [Reducing Diffusion Model Memorization with Higher Order Langevin Dynamics](https://mlsanity.com/paper/2605.19170/) (2026)
- [Filtered Posterior Mean Collections: A Unified Framework for Analytical Models of Diffusion Generalization](https://mlsanity.com/paper/2605.24192/) (2026)

## Abstract (arXiv)

The diffusion probabilistic generative models are widely used to generate high-quality data. Though they can synthetic data that does not exist in the training set, the rationale behind such generalization is still unexplored. In this paper, we formally define the generalization of the generative model, which is measured by the mutual information between the generated data and the training set. The definition originates from the intuition that the model which generates data with less correlation to the training set exhibits better generalization ability. Meanwhile, we show that for the empirical optimal diffusion model, the data generated by a deterministic sampler are all highly related to the training set, thus poor generalization. This result contradicts the observation of the trained diffusion model's (approximating empirical optima) extrapolation ability (generating unseen data). To understand this contradiction, we empirically verify the difference between the sufficiently trained diffusion model and the empirical optima. We found, though obtained through sufficient training, there still exists a slight difference between them, which is critical to making the diffusion model generalizable. Moreover, we propose another training objective whose empirical optimal solution has no potential generalization problem. We empirically show that the proposed training objective returns a similar model to the original one, which further verifies the generalization ability of the trained diffusion model.