mlsanityData to 7 Oct 2026

Singular-limit analysis of gradient descent with noise injection

arXiv 2404.12293 ↗First posted 18 Apr 2024cs.LG math.PRcited by 5

Not on the map. Why?

Abstract (arXiv)

We study the limiting dynamics of a large class of noisy gradient descent systems in the overparameterized regime. In this regime the set of global minimizers of the loss is large, and when initialized in a neighbourhood of this zero-loss set a noisy gradient descent algorithm slowly evolves along this set. In some cases this slow evolution has been related to better generalisation properties. We characterize this evolution for the broad class of noisy gradient descent systems in the limit of small step size. Our results show that the structure of the noise affects not just the form of the limiting process, but also the time scale at which the evolution takes place. We apply the theory to Dropout, label noise and classical SGD (minibatching) noise, and show that these evolve on different two time scales. Classical SGD even yields a trivial evolution on both time scales, implying that additional noise is required for regularization. The results are inspired by the training of neural networks, but the theorems apply to noisy gradient descent of any loss that has a non-trivial zero-loss set.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 38 of its 68 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

38 of 68 references.

References · 38

  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  2. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  3. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  4. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  5. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  6. 2017Three Factors Influencing Minima in SGDcited by 270
  7. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  8. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  9. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  10. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  11. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  12. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
26 more
  1. 2013Dropout Training as Adaptive Regularizationcited by 89
  2. 2020Relative Flatness and Generalizationcited by 88
  3. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 81
  4. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  5. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  6. 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
  7. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  8. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  9. 2022Anticorrelated Noise Injection for Improved Generalizationcited by 41
  10. 2018The loss landscape of overparameterized neural networkscited by 38
  11. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 38
  12. 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 27
  13. 2023Why (and When) does Local SGD Generalize Better than SGD?cited by 26
  14. 2019Mean Field Limit of the Learning Dynamics of Multilayer Neural Networkscited by 25
  15. 2021Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscited by 25
  16. 2022Implicit regularization of dropoutcited by 24
  17. 2018On the Implicit Bias of Dropoutcited by 22
  18. 2020On Convergence and Generalization of Dropout Trainingcited by 19
  19. 2021On minimal representations of shallow ReLU networkscited by 13
  20. 2018Theory of Deep Learning IIb: Optimization Properties of SGDcited by 12
  21. 2019On Dropout and Nuclear Norm Regularizationcited by 12
  22. 2022Weight Expansion: A New Perspective on Dropout and Generalizationcited by 6
  23. 2022Flatter, faster: scaling momentum for optimal speedup of SGDcited by 5
  24. 2020Almost Sure Convergence of Dropout Algorithms for Neural Networkscited by 4
  25. 2020Asymptotic convergence rate of Dropout on shallow linear neural networkscited by 4
  26. 2020Universal Approximation in Dropout Neural Networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2025A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisationcited by 11
  2. 2024How Neural Networks Learn the Support is an Implicit Regularization Effect of SGDcited by 8
  3. 2025Dropout Neural Network Training Viewed from a Percolation Perspectivecited by 2
  4. 2025The Interplay of Statistics and Noisy Optimization: Learning Linear Predictors with Random Data Weightscited by 0
  5. 2026Large Spikes in Stochastic Gradient Descent: A Large-Deviations Viewcited by 0