Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descent
Not on the map. Why?
Abstract (arXiv)
We study the dynamics of a continuous-time model of the Stochastic Gradient Descent (SGD) for the least-square problem. Indeed, pursuing the work of Li et al. (2019), we analyze Stochastic Differential Equations (SDEs) that model SGD either in the case of the training loss (finite samples) or the population one (online setting). A key qualitative feature of the dynamics is the existence of a perfect interpolator of the data, irrespective of the sample size. In both scenarios, we provide precise, non-asymptotic rates of convergence to the (possibly degenerate) stationary distribution. Additionally, we describe this asymptotic distribution, offering estimates of its mean, deviations from it, and a proof of the emergence of heavy-tails related to the step-size magnitude. Numerical simulations supporting our findings are also presented.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 28 of its 50 references that are in the map. First posted on arXiv 2 Jul 2024.
Builds on
28 of 50 references.
References · 28
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
- 2019Benign Overfitting in Linear Regressioncited by 544
- 2021Deep learning: a statistical viewpointcited by 239
- 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
- 2020The Heavy-Tail Phenomenon in SGDcited by 122
- 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
- 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
- 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
- 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
- 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
- 2018A Walk with SGDcited by 74
16 more
- 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
- 2018Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passescited by 69
- 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
- 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
- 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 58
- 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 53
- 2021Last iterate convergence of SGD for Least-Squares in the Interpolation regimecited by 41
- 2020Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Modelcited by 39
- 2021Strength of Minibatch Noise in SGDcited by 39
- 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 38
- 2021Power-law escape rate of SGDcited by 29
- 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
- 2022Homogenization of SGD in high-dimensions: Exact dynamics and generalization propertiescited by 26
- 2021Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscited by 25
- 2016Optimal Rates for Multi-pass Stochastic Gradient Methodscited by 8
- 2024Emergence of heavy tails in homogenized stochastic gradient descentcited by 4
Built on by
Citing papers per year: 2024: 1; 2025: 1; 2026: 1.
3 papers.