# The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications

arXiv: https://arxiv.org/abs/2412.09726 · HTML: https://mlsanity.com/paper/2412.09726/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 15 of its 49 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 15 of 15)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [Generalization in diffusion models arises from geometry-adaptive harmonic representations](https://mlsanity.com/paper/2310.02557/) (2023)
- [PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior](https://mlsanity.com/paper/2106.06406/) (2021)
- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [Diffusion Models Generate Images Like Painters: an Analytical Theory of Outline First, Details Later](https://mlsanity.com/paper/2303.02490/) (2023)
- [Closed-Form Diffusion Models](https://mlsanity.com/paper/2310.12395/) (2023)
- [Learning Mixtures of Gaussians Using Diffusion Models](https://mlsanity.com/paper/2404.18869/) (2024)
- [On the Generalization of Diffusion Model](https://mlsanity.com/paper/2305.14712/) (2023)
- [The Emergence of Reproducibility and Generalizability in Diffusion Models](https://mlsanity.com/paper/2310.05264/) (2023)
- [Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors](https://mlsanity.com/paper/2405.14250/) (2024)
- [The Hidden Linear Structure in Score-Based Models and its Application](https://mlsanity.com/paper/2311.10892/) (2023)
- [Score-based generative models learn manifold-like structures with constrained mixing](https://mlsanity.com/paper/2311.09952/) (2023)

## Built on by (most cited first, 30 of 32)

- [An analytic theory of creativity in convolutional diffusion models](https://mlsanity.com/paper/2412.20292/) (2024)
- [An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models](https://mlsanity.com/paper/2503.03206/) (2025)
- [Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models](https://mlsanity.com/paper/2505.16959/) (2025)
- [Locality in Image Diffusion Models Emerges from Data Statistics](https://mlsanity.com/paper/2509.09672/) (2025)
- [Generalization through variance: how noise shapes inductive biases in diffusion models](https://mlsanity.com/paper/2504.12532/) (2025)
- [On the Edge of Memorization in Diffusion Models](https://mlsanity.com/paper/2508.17689/) (2025)
- [A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective](https://mlsanity.com/paper/2509.16499/) (2025)
- [Selective Underfitting in Diffusion Models](https://mlsanity.com/paper/2510.01378/) (2025)
- [On the Interpolation Effect of Score Smoothing in Diffusion Models](https://mlsanity.com/paper/2502.19499/) (2025)
- [Spectral Analysis of Diffusion Models with Application to Schedule Design](https://mlsanity.com/paper/2502.00180/) (2025)
- [A Random Matrix Theory Perspective on the Consistency of Diffusion Models](https://mlsanity.com/paper/2602.02908/) (2026)
- [Antithetic Noise in Diffusion Models](https://mlsanity.com/paper/2506.06185/) (2025)
- [Geometric Regularity in Deterministic Sampling Dynamics of Diffusion-based Generative Models](https://mlsanity.com/paper/2506.10177/) (2025)
- [Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules](https://mlsanity.com/paper/2510.26997/) (2025)
- [Emergence of Distortions in High-Dimensional Guided Diffusion Models](https://mlsanity.com/paper/2602.00716/) (2026)
- [Score-based generative emulation of impact-relevant Earth system model outputs](https://mlsanity.com/paper/2510.04358/) (2025)
- [Geometry-Aware Discretization Error of Diffusion Models](https://mlsanity.com/paper/2605.08392/) (2026)
- [An exact information theory of generalization phase transitions in Bayesian diffusion models](https://mlsanity.com/paper/2607.08041/) (2026)
- [Dynamical symmetries in the fluctuation-driven regime: an application of Noether's theorem to noisy dynamical systems](https://mlsanity.com/paper/2504.09761/) (2025)
- [Understanding Generalization in Diffusion Distillation via Probability Flow Distance](https://mlsanity.com/paper/2505.20123/) (2025)
- [What Time Is It? How Data Geometry Makes Time Conditioning Optional for Flow Matching](https://mlsanity.com/paper/2605.08344/) (2026)
- [Demystifying Adversarial Robustness in Diffusion Models: Compression, Randomness, and Geometry](https://mlsanity.com/paper/2505.22839/) (2025)
- [GAGA: Gaussianity-Aware Gaussian Approximation for Efficient 3D Molecular Generation](https://mlsanity.com/paper/2507.09043/) (2025)
- [The two clocks and the innovation window: When and how generative models learn rules](https://mlsanity.com/paper/2605.10019/) (2026)
- [Improving Diffusion Generative Models via Truncated Karhunen--Lo`eve Expansion](https://mlsanity.com/paper/2503.17657/) (2025)
- [A Geometric Unification of Generative AI with Manifold-Probabilistic Projection Models](https://mlsanity.com/paper/2510.00666/) (2025)
- [A Variational Manifold Embedding Framework for Nonlinear Dimensionality Reduction](https://mlsanity.com/paper/2511.22128/) (2025)
- [A Gaussian Perspective for Distributional Discrepancy in Generative Diffusion Models](https://mlsanity.com/paper/2601.13602/) (2026)
- [General and Efficient Steering of Diffusion Models](https://mlsanity.com/paper/2602.11395/) (2026)
- [Filtered Posterior Mean Collections: A Unified Framework for Analytical Models of Diffusion Generalization](https://mlsanity.com/paper/2605.24192/) (2026)

## Abstract (arXiv)

By learning the gradient of smoothed data distributions, diffusion models can iteratively generate samples from complex distributions. The learned score function enables their generalization capabilities, but how the learned score relates to the score of the underlying data manifold remains largely unclear. Here, we aim to elucidate this relationship by comparing learned neural scores to the scores of two kinds of analytically tractable distributions: Gaussians and Gaussian mixtures. The simplicity of the Gaussian model makes it theoretically attractive, and we show that it admits a closed-form solution and predicts many qualitative aspects of sample generation dynamics. We claim that the learned neural score is dominated by its linear (Gaussian) approximation for moderate to high noise scales, and supply both theoretical and empirical arguments to support this claim. Moreover, the Gaussian approximation empirically works for a larger range of noise scales than naive theory suggests it should, and is preferentially learned early in training. At smaller noise scales, we observe that learned scores are better described by a coarse-grained (Gaussian mixture) approximation of training data than by the score of the training distribution, a finding consistent with generalization. Our findings enable us to precisely predict the initial phase of trained models' sampling trajectories through their Gaussian approximations. We show that this allows the skipping of the first 15-30% of sampling steps while maintaining high sample quality (with a near state-of-the-art FID score of 1.93 on CIFAR-10 unconditional generation). This forms the foundation of a novel hybrid sampling method, termed analytical teleportation, which can seamlessly integrate with and accelerate existing samplers, including DPM-Solver-v3 and UniPC. Our findings suggest ways to improve the design and training of diffusion models.