# An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models

arXiv: https://arxiv.org/abs/2503.03206 · HTML: https://mlsanity.com/paper/2503.03206/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 17 of its 64 references that are in the map.
- First posted on arXiv 5 Mar 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 17 of 17)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Flow Matching Guide and Code](https://mlsanity.com/paper/2412.06264/) (2024)
- [Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models](https://mlsanity.com/paper/2410.10733/) (2024)
- [Generative Modelling With Inverse Heat Dissipation](https://mlsanity.com/paper/2206.13397/) (2022)
- [The Low-Rank Simplicity Bias in Deep Networks](https://mlsanity.com/paper/2103.10427/) (2021)
- [An analytic theory of creativity in convolutional diffusion models](https://mlsanity.com/paper/2412.20292/) (2024)
- [A Unifying View on Implicit Bias in Training Linear Neural Networks](https://mlsanity.com/paper/2010.02501/) (2020)
- [Diffusion Models Generate Images Like Painters: an Analytical Theory of Outline First, Details Later](https://mlsanity.com/paper/2303.02490/) (2023)
- [Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generators](https://mlsanity.com/paper/1910.14634/) (2019)
- [Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure](https://mlsanity.com/paper/2410.24060/) (2024)
- [The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications](https://mlsanity.com/paper/2412.09726/) (2024)
- [The Spectral Bias of the Deep Image Prior](https://mlsanity.com/paper/1912.08905/) (2019)
- [Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors](https://mlsanity.com/paper/2405.14250/) (2024)
- [The Hidden Linear Structure in Score-Based Models and its Application](https://mlsanity.com/paper/2311.10892/) (2023)
- [Blink of an eye: a simple theory for feature localization in generative models](https://mlsanity.com/paper/2502.00921/) (2025)

## Built on by (most cited first, 26 of 26)

- [Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architectures](https://mlsanity.com/paper/2512.20607/) (2025)
- [Generalization Dynamics of Linear Diffusion Models](https://mlsanity.com/paper/2505.24769/) (2025)
- [What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion](https://mlsanity.com/paper/2605.07915/) (2026)
- [Diffusing in the Right Space: A Systematic Study of Latent Diffusability](https://mlsanity.com/paper/2606.03578/) (2026)
- [A theory of learning data statistics in diffusion models, from easy to hard](https://mlsanity.com/paper/2603.12901/) (2026)
- [Two Calm Ends and the Wild Middle: A Geometric Picture of Memorization in Diffusion Models](https://mlsanity.com/paper/2602.17846/) (2026)
- [The Interplay of Data Structure and Imbalance in the Learning Dynamics of Diffusion Models](https://mlsanity.com/paper/2605.06367/) (2026)
- [Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors](https://mlsanity.com/paper/2606.06179/) (2026)
- [When Marginals Match but Structure Fails: Covariance Fidelity in Generative Models](https://mlsanity.com/paper/2603.17041/) (2026)
- [Local Intrinsic Dimension Unveils Hallucinations in Diffusion Models](https://mlsanity.com/paper/2605.05026/) (2026)
- [The two clocks and the innovation window: When and how generative models learn rules](https://mlsanity.com/paper/2605.10019/) (2026)
- [The critical slowing down in training diffusion models](https://mlsanity.com/paper/2605.12597/) (2026)
- [Memorisation, convergence and generalisation in generative models](https://mlsanity.com/paper/2605.21402/) (2026)
- [Where the Score Lives: A Wavelet View of Diffusion](https://mlsanity.com/paper/2606.08309/) (2026)
- [Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles](https://mlsanity.com/paper/2606.09718/) (2026)
- [SALD: Self-Referenced Advantage Learning for Diffusion Models](https://mlsanity.com/paper/2610.01496/) (2026)
- [Sparsely Supervised Diffusion](https://mlsanity.com/paper/2602.02699/) (2026)
- [ScatterPrism: convergence for generative simulation and inverse problems in particle and nuclear physics](https://mlsanity.com/paper/2604.01313/) (2026)
- [Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMs](https://mlsanity.com/paper/2604.10074/) (2026)
- [Spectral Energy Centroid: a Metric for Improving Performance and Analyzing Spectral Bias in Implicit Neural Representations](https://mlsanity.com/paper/2605.12709/) (2026)
- [Provably Learning Diffusion Models under the Manifold Hypothesis: Collapse and Refine](https://mlsanity.com/paper/2605.20235/) (2026)
- [Order within Chaos: Capturing Intrinsic Energy Anomalies for AI-Manipulated Image Forgery Localization](https://mlsanity.com/paper/2606.02178/) (2026)
- [Breaking the Lock-in: Diversifying Text-to-Image Generation via Representation Modulation](https://mlsanity.com/paper/2606.06813/) (2026)
- [Can Deep Generative Models Reproduce Non-Stationary Gaussian Random Fields?](https://mlsanity.com/paper/2607.25929/) (2026)
- [Weighting Schedules Govern What and When Score-Based Generative Models Learn from Multimodal Data](https://mlsanity.com/paper/2609.35322/) (2026)
- [First Learn, Then Memorize: The Spectral Bias of Diffusion Models](https://mlsanity.com/paper/2609.35377/) (2026)

## Abstract (arXiv)

We develop an analytical framework for understanding how the generated distribution evolves during diffusion model training. Leveraging a Gaussian-equivalence principle, we solve the full-batch gradient-flow dynamics of linear and convolutional denoisers and integrate the resulting probability-flow ODE, yielding analytic expressions for the generated distribution. The theory exposes a universal inverse-variance spectral law: the time for an eigen- or Fourier mode to match its target variance scales as τproptoλ-1, so high-variance (coarse) structure is mastered orders of magnitude sooner than low-variance (fine) detail. Extending the analysis to deep linear networks and circulant full-width convolutions shows that weight sharing merely multiplies learning rates -- accelerating but not eliminating the bias -- whereas local convolution introduces a qualitatively different bias. Experiments on Gaussian and natural-image datasets confirm the spectral law persists in deep MLP-based UNet. Convolutional U-Nets, however, display rapid near-simultaneous emergence of many modes, implicating local convolution in reshaping learning dynamics. These results underscore how data covariance governs the order and speed with which diffusion models learn, and they call for deeper investigation of the unique inductive biases introduced by local convolution.