Effective continuous equations for adaptive SGD: a stochastic analysis view
Not on the map. Why?
Abstract (arXiv)
We present a theoretical analysis of some popular adaptive Stochastic Gradient Descent (SGD) methods in the small learning rate regime. Using the stochastic modified equations framework introduced by Li et al., we derive effective continuous stochastic dynamics for these methods. Our key contribution is that sampling-induced noise in SGD manifests in the limit as independent Brownian motions driving the parameter and gradient second momentum evolutions. Furthermore, extending the approach of Malladi et al., we investigate scaling rules between the learning rate and key hyperparameters in adaptive methods, characterising all non-trivial limiting dynamics.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 25 Sep 2025.
Builds on
27 of 39 references.
References · 27
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2016An overview of gradient descent optimization algorithmscited by 1,033
- 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
- 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
- 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
- 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
- 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
- 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
- 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
- 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
- 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
- 2020Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentumcited by 40
15 more
- 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 39
- 2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 33
- 2021The effective noise of Stochastic Gradient Descentcited by 25
- 2024Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning ratescited by 15
- 2024Convergence rates for the Adam optimizercited by 15
- 2023Stochastic Modified Equations and Dynamics of Dropout Algorithmcited by 11
- 2023Exponential convergence rates for momentum stochastic gradient descent in the overparametrized settingcited by 8
- 2024Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activationcited by 8
- 2025Sharp higher order convergence rates for the Adam optimizercited by 8
- 2023Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descentcited by 7
- 2025Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networkscited by 6
- 2018Differential Equations for Modeling Asynchronous Algorithmscited by 4
- 2025Averaged Adam accelerates stochastic optimization in the training of deep neural network approximations for partial differential equation and optimal control problemscited by 3
- 2024Stochastic Modified Flows for Riemannian Stochastic Gradient Descentcited by 2
- 2024A Hessian-Aware Stochastic Differential Equation for Modelling SGDcited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.