# First Learn, Then Memorize: The Spectral Bias of Diffusion Models

arXiv: https://arxiv.org/abs/2609.35377 · HTML: https://mlsanity.com/paper/2609.35377/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 20 of its 85 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 20 of 20)

- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [The Gaussian equivalence of generative models for learning with shallow neural networks](https://mlsanity.com/paper/2006.14709/) (2020)
- [On the Generalization Properties of Diffusion Models](https://mlsanity.com/paper/2311.01797/) (2023)
- [Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression](https://mlsanity.com/paper/2204.10425/) (2022)
- [Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization](https://mlsanity.com/paper/2401.15604/) (2024)
- [Memorization and Regularization in Generative Diffusion Models](https://mlsanity.com/paper/2501.15785/) (2025)
- [A Good Score Does not Lead to A Good Generative Model](https://mlsanity.com/paper/2401.04856/) (2024)
- [Losing dimensions: Geometric memorization in generative diffusion](https://mlsanity.com/paper/2410.08727/) (2024)
- [An Equivalence Principle for the Spectrum of Random Inner-Product Kernel Matrices with Polynomial Scalings](https://mlsanity.com/paper/2205.06308/) (2022)
- [An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models](https://mlsanity.com/paper/2503.03206/) (2025)
- [Denoising Score Matching with Random Features: Insights on Diffusion Models from Precise Learning Curves](https://mlsanity.com/paper/2502.00336/) (2025)
- [Asymptotics of Non-Convex Generalized Linear Models in High-Dimensions: A proof of the replica formula](https://mlsanity.com/paper/2502.20003/) (2025)
- [Universality of Kernel Random Matrices and Kernel Regression in the Quadratic Regime](https://mlsanity.com/paper/2408.01062/) (2024)
- [Kernel ridge regression under power-law data: spectrum and generalization](https://mlsanity.com/paper/2510.04780/) (2025)
- [Generalization Dynamics of Linear Diffusion Models](https://mlsanity.com/paper/2505.24769/) (2025)
- [Local Coverage Governs Memorization in Diffusion Models](https://mlsanity.com/paper/2606.14390/) (2026)
- [Biased Generalization in Diffusion Models](https://mlsanity.com/paper/2603.03469/) (2026)
- [Generalization, memorization, and overfitting for diffusion models trained in the lazy high-dimensional regime](https://mlsanity.com/paper/2608.23938/) (2026)

## Abstract (arXiv)

Diffusion models trained on a finite dataset first learn to generate novel, high-quality samples and only much later collapse onto their training set. We identify the mechanism behind this separation of timescales and the object that probes it. The training dynamics of the score function are governed---exactly, and at any width---by the Gram matrix of the Neural Tangent Kernel (NTK) evaluated on the noisy training data, so the timescales of generalization and of memorization must be encoded in its spectrum. We show that they are, and that the structure responsible has no analogue in standard kernel settings. The use of multiple noise realizations per sample (m noised copies at a fixed noise level) in the score-matching loss is what restructures the Gram matrix spectrum into two distinct parts. The first, of large eigenvalues, carries the global features of the target distribution and is present already for m=1. The second, which the repeated noising creates, consists of the smallest eigenvalues and is supported on eigenvectors aligned with the sample-specific noise directions; it sets a memorization timescale parametrically larger in the training set size n. We establish this picture on two fronts. Analytically, we solve the spectrum in the lazy high-dimensional limit for both linear (n asymp d) and polynomial (n asymp dk) sample complexities, and prove through a bias--variance decomposition that the first bulk minimizes the approximation error while the second drives the error associated with memorization. Empirically, we show the same two-bulk structure in Convolutional NTKs on CelebA and in finite-width U-Nets trained well beyond the lazy regime, and we make the link causal: truncating the Gram matrix at rank r tunes the generalization--memorization transition, and an L2 penalty targeting the second bulk suppresses memorization in feature-learning U-Nets.