# Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization

arXiv: https://arxiv.org/abs/2502.03435 · HTML: https://mlsanity.com/paper/2502.03435/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 39 of its 52 references that are in the map.
- First posted on arXiv 5 Feb 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 39)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [SegDiff: Image Segmentation with Diffusion Probabilistic Models](https://mlsanity.com/paper/2112.00390/) (2021)
- [Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localization](https://mlsanity.com/paper/2308.03686/) (2023)
- [Understanding and Mitigating Copying in Diffusion Models](https://mlsanity.com/paper/2305.20086/) (2023)
- [Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models](https://mlsanity.com/paper/2208.09399/) (2022)
- [Generalization in diffusion models arises from geometry-adaptive harmonic representations](https://mlsanity.com/paper/2310.02557/) (2023)
- [Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks](https://mlsanity.com/paper/1907.04595/) (2019)
- [Dynamical Regimes of Diffusion Models](https://mlsanity.com/paper/2402.18491/) (2024)
- [On Memorization in Diffusion Models](https://mlsanity.com/paper/2310.02664/) (2023)
- [Ambient Diffusion: Learning Clean Distributions from Corrupted Data](https://mlsanity.com/paper/2305.19256/) (2023)
- [Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach](https://mlsanity.com/paper/1805.00915/) (2018)
- [Step-unrolled Denoising Autoencoders for Text Generation](https://mlsanity.com/paper/2112.06749/) (2021)
- [On Provable Copyright Protection for Generative Models](https://mlsanity.com/paper/2302.10870/) (2023)
- [Spontaneous Symmetry Breaking in Generative Diffusion Models](https://mlsanity.com/paper/2305.19693/) (2023)
- [SGD with Large Step Sizes Learns Sparse Features](https://mlsanity.com/paper/2210.05337/) (2022)
- [Memorization and Regularization in Generative Diffusion Models](https://mlsanity.com/paper/2501.15785/) (2025)
- [Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure](https://mlsanity.com/paper/2410.24060/) (2024)
- [A Good Score Does not Lead to A Good Generative Model](https://mlsanity.com/paper/2401.04856/) (2024)

## Built on by (most cited first, 17 of 17)

- [Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training](https://mlsanity.com/paper/2505.17638/) (2025)
- [On the Edge of Memorization in Diffusion Models](https://mlsanity.com/paper/2508.17689/) (2025)
- [Selective Underfitting in Diffusion Models](https://mlsanity.com/paper/2510.01378/) (2025)
- [Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusion](https://mlsanity.com/paper/2502.21278/) (2025)
- [Generalization of Diffusion Models Arises with a Balanced Representation Space](https://mlsanity.com/paper/2512.20963/) (2025)
- [Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regression](https://mlsanity.com/paper/2506.02336/) (2025)
- [The Generation Phases of Flow Matching: a Denoising Perspective](https://mlsanity.com/paper/2510.24830/) (2025)
- [Diffusion Model's Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifold](https://mlsanity.com/paper/2602.06021/) (2026)
- [Local Coverage Governs Memorization in Diffusion Models](https://mlsanity.com/paper/2606.14390/) (2026)
- [Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stability](https://mlsanity.com/paper/2606.18080/) (2026)
- [Understanding diffusion models requires rethinking (again) generalization](https://mlsanity.com/paper/2605.06077/) (2026)
- [Convergence of optimizers implies eigenvalues filtering at equilibrium](https://mlsanity.com/paper/2510.09034/) (2025)
- [Adynamical systems view of training generativemodels and the memorization phenomenon](https://mlsanity.com/paper/2605.19483/) (2026)
- [A Data-Interventional Framework for Auditing Privacy and Fairness in Generative Medical Imaging](https://mlsanity.com/paper/2609.26623/) (2026)
- [Background Gradients Shape Memorization in Flow Matching](https://mlsanity.com/paper/2609.33210/) (2026)
- [Controlling Polar Exposure to Delay Memorization in Diffusion Models](https://mlsanity.com/paper/2610.02780/) (2026)
- [Memorization and Malign Generalization in Conditional Diffusion Models with Random Features](https://mlsanity.com/paper/2610.11288/) (2026)

## Abstract (arXiv)

Denoising score matching plays a pivotal role in the performance of diffusion-based generative models. However, the empirical optimal score--the exact solution to the denoising score matching--leads to memorization, where generated samples replicate the training data. Yet, in practice, only a moderate degree of memorization is observed, even without explicit regularization. In this paper, we investigate this phenomenon by uncovering an implicit regularization mechanism driven by large learning rates. Specifically, we show that in the small-noise regime, the empirical optimal score exhibits high irregularity. We then prove that, when trained by stochastic gradient descent with a large enough learning rate, neural networks cannot stably converge to a local minimum with arbitrarily small excess risk. Consequently, the learned score cannot be arbitrarily close to the empirical optimal score, thereby mitigating memorization. To make the analysis tractable, we consider one-dimensional data and two-layer neural networks. Experiments validate the crucial role of the learning rate in preventing memorization, even beyond the one-dimensional setting.