mlsanityData to 7 Oct 2026

Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization

arXiv 2502.03435 ↗First posted 5 Feb 2025stat.ML cs.LGcited by 16

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Denoising score matching plays a pivotal role in the performance of diffusion-based generative models. However, the empirical optimal score--the exact solution to the denoising score matching--leads to memorization, where generated samples replicate the training data. Yet, in practice, only a moderate degree of memorization is observed, even without explicit regularization. In this paper, we investigate this phenomenon by uncovering an implicit regularization mechanism driven by large learning rates. Specifically, we show that in the small-noise regime, the empirical optimal score exhibits high irregularity. We then prove that, when trained by stochastic gradient descent with a large enough learning rate, neural networks cannot stably converge to a local minimum with arbitrarily small excess risk. Consequently, the learned score cannot be arbitrarily close to the empirical optimal score, thereby mitigating memorization. To make the analysis tractable, we consider one-dimensional data and two-layer neural networks. Experiments validate the crucial role of the learning rate in preventing memorization, even beyond the one-dimensional setting.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 39 of its 52 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

39 of 52 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Memorization and Regularization in Generative Diffusion Models27 Jan
  2. 2024Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure31 Oct
  3. 2024Dynamical Regimes of Diffusion Models28 Feb
  4. 2024A Good Score Does not Lead to A Good Generative Model10 Jan
  5. 2023On Memorization in Diffusion Models4 Oct
  6. 2023Generalization in diffusion models arises from geometry-adaptive harmonic representations4 Oct
  7. 2023Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localization7 Aug
  8. 2023Understanding and Mitigating Copying in Diffusion Models31 May
  9. 2023Spontaneous Symmetry Breaking in Generative Diffusion Models31 May
  10. 2023On the Generalization of Diffusion Model24 May
  11. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  12. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
1 more
  1. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 26

  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2023Extracting Training Data from Diffusion Modelscited by 723
  5. 2018On Lazy Training in Differentiable Programmingcited by 671
  6. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  7. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  8. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  9. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  10. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
  11. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Modelscited by 279
  12. 2022Diffusion-based Time Series Imputation and Forecasting with Structured State Space Modelscited by 195
14 more
  1. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  2. 2023Ambient Diffusion: Learning Clean Distributions from Corrupted Datacited by 108
  3. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  4. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  5. 2023On Provable Copyright Protection for Generative Modelscited by 97
  6. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  7. 2024Denoising: A Powerful Building-Block for Imaging, Inverse Problems, and Machine Learningcited by 35
  8. 2023On Copyright Risks of Text-to-Image Diffusion Modelscited by 20
  9. 2023Penalising the biases in norm regularisation enforces sparsitycited by 17
  10. 2023The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networkscited by 15
  11. 2019Online Regularization by Denoising with Applications to Phase Retrievalcited by 12
  12. 2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
  13. 2023How do Minimum-Norm Shallow Denoisers Look in Function Space?cited by 9
  14. 2016Image Restoration and Reconstruction using Variable Splitting and Class-adapted Image Priorscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 8.

16 papers.

Sort:

By area:

  1. 2025Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Trainingcited by 78
  2. 2025On the Edge of Memorization in Diffusion Modelscited by 22
  3. 2025Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusioncited by 17
  4. 2025Selective Underfitting in Diffusion Modelscited by 17
  5. 2025Generalization of Diffusion Models Arises with a Balanced Representation Spacecited by 10
  6. 2025Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regressioncited by 7
  7. 2025The Generation Phases of Flow Matching: a Denoising Perspectivecited by 6
  8. 2026Diffusion Model's Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifoldcited by 5
  9. 2026Local Coverage Governs Memorization in Diffusion Modelscited by 5
  10. 2026Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 3
  11. 2026Understanding diffusion models requires rethinking (again) generalizationcited by 2
  12. 2025Convergence of optimizers implies eigenvalues filtering at equilibriumcited by 0
4 more
  1. 2026Adynamical systems view of training generativemodels and the memorization phenomenoncited by 0
  2. 2026A Data-Interventional Framework for Auditing Privacy and Fairness in Generative Medical Imagingcited by 0
  3. 2026Background Gradients Shape Memorization in Flow Matchingcited by 0
  4. 2026Memorization and Malign Generalization in Conditional Diffusion Models with Random Featurescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Denoising Score Matching with Random Features: Insights on Diffusion Models from Precise Learning Curves
  2. 2026Generalization, memorization, and overfitting for diffusion models trained in the lazy high-dimensional regime
  3. 2025How Diffusion Models Memorize
  4. 2026Memorization Control in Diffusion Models from Denoising-centric Perspective
  5. 2025Optimizing Input of Denoising Score Matching is Biased Towards Higher Score Norm
  6. 2025Provable Separations between Memorization and Generalization in Diffusion Models
  7. 2025Implicit Regularisation in Diffusion Models: An Algorithm-Dependent Generalisation Analysis
  8. 2026Detecting and Mitigating Memorization in Diffusion Models through Anisotropy of the Log-Probability