mlsanityData to 14 Oct 2026

An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models

arXiv 2503.03206 ↗First posted 5 Mar 2025cs.LG cs.CV math.STcited by 26

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

We develop an analytical framework for understanding how the generated distribution evolves during diffusion model training. Leveraging a Gaussian-equivalence principle, we solve the full-batch gradient-flow dynamics of linear and convolutional denoisers and integrate the resulting probability-flow ODE, yielding analytic expressions for the generated distribution. The theory exposes a universal inverse-variance spectral law: the time for an eigen- or Fourier mode to match its target variance scales as τproptoλ-1, so high-variance (coarse) structure is mastered orders of magnitude sooner than low-variance (fine) detail. Extending the analysis to deep linear networks and circulant full-width convolutions shows that weight sharing merely multiplies learning rates -- accelerating but not eliminating the bias -- whereas local convolution introduces a qualitatively different bias. Experiments on Gaussian and natural-image datasets confirm the spectral law persists in deep MLP-based UNet. Convolutional U-Nets, however, display rapid near-simultaneous emergence of many modes, implicating local convolution in reshaping learning dynamics. These results underscore how data covariance governs the order and speed with which diffusion models learn, and they call for deeper investigation of the unique inductive biases introduced by local convolution.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 17 of its 64 references that are in the map. First posted on arXiv 5 Mar 2025.

Builds on

17 of 64 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications12 Dec
  2. 2024Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure31 Oct
  3. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models14 Oct
  4. 2024Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors23 May
  5. 2023The Hidden Linear Structure in Score-Based Models and its Application17 Nov
  6. 2023Diffusion Models Generate Images Like Painters: an Analytical Theory of Outline First, Details Later4 Mar
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun

Other references · 10

  1. 2022Flow Matching for Generative Modelingcited by 5,702
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  3. 2024Flow Matching Guide and Codecited by 301
  4. 2022Generative Modelling With Inverse Heat Dissipationcited by 167
  5. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  6. 2024An analytic theory of creativity in convolutional diffusion modelscited by 104
  7. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
  8. 2019Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generatorscited by 40
  9. 2019The Spectral Bias of the Deep Image Priorcited by 21
  10. 2025Blink of an eye: a simple theory for feature localization in generative modelscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 24.

26 papers.

Sort:

By area:

  1. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  2. 2025Generalization Dynamics of Linear Diffusion Modelscited by 9
  3. 2026What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusioncited by 9
  4. 2026Diffusing in the Right Space: A Systematic Study of Latent Diffusabilitycited by 6
  5. 2026A theory of learning data statistics in diffusion models, from easy to hardcited by 5
  6. 2026Two Calm Ends and the Wild Middle: A Geometric Picture of Memorization in Diffusion Modelscited by 3
  7. 2026The Interplay of Data Structure and Imbalance in the Learning Dynamics of Diffusion Modelscited by 2
  8. 2026Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errorscited by 2
  9. 2026When Marginals Match but Structure Fails: Covariance Fidelity in Generative Modelscited by 1
  10. 2026Local Intrinsic Dimension Unveils Hallucinations in Diffusion Modelscited by 1
  11. 2026The two clocks and the innovation window: When and how generative models learn rulescited by 1
  12. 2026The critical slowing down in training diffusion modelscited by 1
14 more
  1. 2026Memorisation, convergence and generalisation in generative modelscited by 1
  2. 2026Where the Score Lives: A Wavelet View of Diffusioncited by 1
  3. 2026Evaluating the Representation Space of Diffusion Models via Self-Supervised Principlescited by 1
  4. 2026SALD: Self-Referenced Advantage Learning for Diffusion Modelscited by 1
  5. 2026Sparsely Supervised Diffusioncited by 0
  6. 2026ScatterPrism: convergence for generative simulation and inverse problems in particle and nuclear physicscited by 0
  7. 2026Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMscited by 0
  8. 2026Spectral Energy Centroid: a Metric for Improving Performance and Analyzing Spectral Bias in Implicit Neural Representationscited by 0
  9. 2026Provably Learning Diffusion Models under the Manifold Hypothesis: Collapse and Refinecited by 0
  10. 2026Order within Chaos: Capturing Intrinsic Energy Anomalies for AI-Manipulated Image Forgery Localizationcited by 0
  11. 2026Breaking the Lock-in: Diversifying Text-to-Image Generation via Representation Modulationcited by 0
  12. 2026Can Deep Generative Models Reproduce Non-Stationary Gaussian Random Fields?cited by 0
  13. 2026Weighting Schedules Govern What and When Score-Based Generative Models Learn from Multimodal Datacited by 0
  14. 2026First Learn, Then Memorize: The Spectral Bias of Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Unmasking Bias in Diffusion Model Training
  2. 2025Diffusion Models: A Mathematical Introduction
  3. 2025Learning What Matters: Steering Diffusion via Spectrally Anisotropic Forward Noise
  4. 2025Generalization through variance: how noise shapes inductive biases in diffusion models
  5. 2025Shaping Inductive Bias in Diffusion Models through Frequency-Based Noise Control
  6. 2026Revisiting Spectral Representations in Generative Diffusion Models
  7. 2025When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Nets
  8. 2024Towards a Mechanistic Explanation of Diffusion Model Generalization