Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Features
Not on the map. Why?
Abstract (arXiv)
We investigate the test risk of continuous-time stochastic gradient flow dynamics in learning theory. Using a path integral formulation we provide, in the regime of a small learning rate, a general formula for computing the difference between test risk curves of pure gradient and stochastic gradient flows. We apply the general theory to a simple model of weak features, which displays the double descent phenomenon, and explicitly compute the corrections brought about by the added stochastic term in the dynamics, as a function of time and model parameters. The analytical results are compared to simulations of discrete-time stochastic gradient descent and show good agreement.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 12 Feb 2024.
Builds on
26 of 43 references.
References · 26
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
- 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
- 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
- 2018Gradient Descent Happens in a Tiny Subspacecited by 217
- 2019Two models of double descent for weak featurescited by 195
- 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
- 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
- 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
- 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
- 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 70
- 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
- 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 59
14 more
- 2016A Variational Analysis of Stochastic Gradient Algorithmscited by 57
- 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
- 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
- 2021Power-law escape rate of SGDcited by 29
- 2022Homogenization of SGD in high-dimensions: Exact dynamics and generalization propertiescited by 27
- 2021The effective noise of Stochastic Gradient Descentcited by 25
- 2021Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problemcited by 23
- 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
- 2021Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature modelcited by 17
- 2021Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic Modelscited by 16
- 2022On the Power-Law Hessian Spectrums in Deep Learningcited by 16
- 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 15
- 2017Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigmscited by 5
- 2023Unveiling the Hessian's Connection to the Decision Boundarycited by 5
Built on by
Citing papers per year: 2025: 1.
1 paper.