# Generalization through variance: how noise shapes inductive biases in diffusion models

arXiv: https://arxiv.org/abs/2504.12532 · HTML: https://mlsanity.com/paper/2504.12532/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 48 of its 55 references that are in the map.
- First posted on arXiv 16 Apr 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 48)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modeling](https://mlsanity.com/paper/2106.01357/) (2021)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions](https://mlsanity.com/paper/2209.11215/) (2022)
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [Sliced Score Matching: A Scalable Approach to Density and Score Estimation](https://mlsanity.com/paper/1905.07088/) (2019)
- [Kernel and Rich Regimes in Overparametrized Models](https://mlsanity.com/paper/2002.09277/) (2020)
- [Convergence of denoising diffusion models under the manifold hypothesis](https://mlsanity.com/paper/2208.05314/) (2022)
- [Score-Based Generative Modeling with Critically-Damped Langevin Diffusion](https://mlsanity.com/paper/2112.07068/) (2021)
- [Understanding and Mitigating Copying in Diffusion Models](https://mlsanity.com/paper/2305.20086/) (2023)
- [Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models](https://mlsanity.com/paper/2306.04675/) (2023)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [Generalization in diffusion models arises from geometry-adaptive harmonic representations](https://mlsanity.com/paper/2310.02557/) (2023)
- [On the Inductive Bias of Neural Tangent Kernels](https://mlsanity.com/paper/1905.12173/) (2019)
- [Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data](https://mlsanity.com/paper/2302.07194/) (2023)

## Built on by (most cited first, 23 of 23)

- [On the Closed-Form of Flow Matching: Generalization Does Not Arise from Target Stochasticity](https://mlsanity.com/paper/2506.03719/) (2025)
- [Memorization to Generalization: Emergence of Diffusion Models from Associative Memory](https://mlsanity.com/paper/2505.21777/) (2025)
- [On the Edge of Memorization in Diffusion Models](https://mlsanity.com/paper/2508.17689/) (2025)
- [Diffusion Models and the Manifold Hypothesis: Log-Domain Smoothing is Geometry Adaptive](https://mlsanity.com/paper/2510.02305/) (2025)
- [Selective Underfitting in Diffusion Models](https://mlsanity.com/paper/2510.01378/) (2025)
- [Provable Separations between Memorization and Generalization in Diffusion Models](https://mlsanity.com/paper/2511.03202/) (2025)
- [UALM: Unified Audio Language Model for Understanding, Generation and Reasoning](https://mlsanity.com/paper/2510.12000/) (2025)
- [The Generation Phases of Flow Matching: a Denoising Perspective](https://mlsanity.com/paper/2510.24830/) (2025)
- [Diffusion Model's Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifold](https://mlsanity.com/paper/2602.06021/) (2026)
- [Deeper Diffusion Models Amplify Bias](https://mlsanity.com/paper/2505.17560/) (2025)
- [Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules](https://mlsanity.com/paper/2510.26997/) (2025)
- [SURFing to the Fundamental Limit of Jet Tagging](https://mlsanity.com/paper/2511.15779/) (2025)
- [Two Calm Ends and the Wild Middle: A Geometric Picture of Memorization in Diffusion Models](https://mlsanity.com/paper/2602.17846/) (2026)
- [Review of Hallucination Understanding in Large Language and Vision Models](https://mlsanity.com/paper/2510.00034/) (2025)
- [Diffusion Models Memorize in Training -- and Generalize in Inference](https://mlsanity.com/paper/2603.13419/) (2026)
- [Matching the Optimal Denoiser in Point Cloud Diffusion with (Improved) Rotational Alignment](https://mlsanity.com/paper/2510.03335/) (2025)
- [Is Your Conditional Diffusion Model Actually Denoising?](https://mlsanity.com/paper/2512.18736/) (2025)
- [The two clocks and the innovation window: When and how generative models learn rules](https://mlsanity.com/paper/2605.10019/) (2026)
- [What Do You Need for Compositional Generalization in Diffusion Planning?](https://mlsanity.com/paper/2505.18083/) (2025)
- [Let Physics Guide Your Protein Flows: Topology-aware Unfolding and Generation](https://mlsanity.com/paper/2509.25379/) (2025)
- [A Variational Manifold Embedding Framework for Nonlinear Dimensionality Reduction](https://mlsanity.com/paper/2511.22128/) (2025)
- [Filtered Posterior Mean Collections: A Unified Framework for Analytical Models of Diffusion Generalization](https://mlsanity.com/paper/2605.24192/) (2026)
- [Joint Flow Matching for Generator-Consistent Classification](https://mlsanity.com/paper/2607.23946/) (2026)

## Abstract (arXiv)

How diffusion models generalize beyond their training set is not known, and is somewhat mysterious given two facts: the optimum of the denoising score matching (DSM) objective usually used to train diffusion models is the score function of the training distribution; and the networks usually used to learn the score function are expressive enough to learn this score to high accuracy. We claim that a certain feature of the DSM objective -- the fact that its target is not the training distribution's score, but a noisy quantity only equal to it in expectation -- strongly impacts whether and to what extent diffusion models generalize. In this paper, we develop a mathematical theory that partly explains this 'generalization through variance' phenomenon. Our theoretical analysis exploits a physics-inspired path integral approach to compute the distributions typically learned by a few paradigmatic under- and overparameterized diffusion models. We find that the distributions diffusion models effectively learn to sample from resemble their training distributions, but with 'gaps' filled in, and that this inductive bias is due to the covariance structure of the noisy target used during training. We also characterize how this inductive bias interacts with feature-related inductive biases.