A note on diffusion limits for stochastic gradient descent
Not on the map. Why?
Abstract (arXiv)
In the machine learning literature stochastic gradient descent has recently been widely discussed for its purported implicit regularization properties. Much of the theory, that attempts to clarify the role of noise in stochastic gradient algorithms, has widely approximated stochastic gradient descent by a stochastic differential equation with Gaussian noise. We provide a novel rigorous theoretical justification for this practice that showcases how the Gaussianity of the noise arises naturally.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 11 of its 26 references that are in the map. First posted on arXiv 20 Oct 2022.
Builds on
11 of 26 references.
References · 11
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
- 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
- 2019SGD: General Analysis and Improved Ratescited by 207
- 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
- 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
- 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 109
- 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 95
- 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
- 2019First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noisecited by 44
- 2017Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigmscited by 5
Built on by
Citing papers per year: 2025: 1.
1 paper.