mlsanityData to 7 Oct 2026

Exponential convergence rates for momentum stochastic gradient descent in the overparametrized setting

arXiv 2302.03550 ↗First posted 7 Feb 2023math.OC cs.LG math.PRcited by 7

Not on the map. Why?

Abstract (arXiv)

We prove explicit bounds on the exponential rate of convergence for the momentum stochastic gradient descent scheme (MSGD) for arbitrary, fixed hyperparameters (learning rate, friction parameter) and its continuous-in-time counterpart in the context of non-convex optimization. In the small step-size regime and in the case of flat minima or large noise intensities, these bounds prove faster convergence of MSGD compared to plain stochastic gradient descent (SGD). The results are shown for objective functions satisfying a local Polyak-Lojasiewicz inequality and under assumptions on the variance of MSGD that are satisfied in overparametrized settings. Moreover, we analyze the optimal choice of the friction parameter and show that the MSGD process almost surely converges to a local minimum.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 22 of its 49 references that are in the map. First posted on arXiv 7 Feb 2023.

Builds on

22 of 49 references.

References · 22

  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  2. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  3. 2020Better Theory for SGD in the Nonconvex Worldcited by 156
  4. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  5. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
  6. 2014Analysis and Design of Optimization Algorithms via Integral Quadratic Constraintscited by 105
  7. 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
  8. 2014Global convergence of the Heavy-ball method for convex optimizationcited by 60
  9. 2017Momentum and Stochastic Momentum for Stochastic Gradient, Newton, Proximal Point and Subspace Descent Methodscited by 49
  10. 2016Stochastic Heavy Ballcited by 40
  11. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 38
  12. 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 27
10 more
  1. 2021Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscited by 25
  2. 2020Asymptotic study of stochastic adaptive algorithm in non-convex landscapecited by 16
  3. 2017Linearly convergent stochastic heavy ball method for minimizing generalization errorcited by 13
  4. 2021On minimal representations of shallow ReLU networkscited by 13
  5. 2021Convergence rates for the Heavy-Ball continuous dynamics for non-convex optimization, under Polyak-L ojasiewicz conditioncited by 11
  6. 2018Non-monotone Behavior of the Heavy Ball Methodcited by 8
  7. 2022Conservative SPDEs as fluctuating mean field limits of stochastic gradient descentcited by 3
  8. 2022Mean-field analysis for heavy ball methods: Dropout-stability, connectivity, and global convergencecited by 3
  9. 2022Gradient-Type Methods for Optimization Problems with Polyak-Lojasiewicz Condition: Early Stopping and Adaptivity to Inexactness Parametercited by 2
  10. 2018Finite-time stability for differential inclusions with applications to neural networkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 3; 2026: 1.

7 papers.

  1. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  2. 2025ODE approximation for the Adam algorithm: General and overparametrized settingcited by 8
  3. 2023Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descentcited by 7
  4. 2025Effective continuous equations for adaptive SGD: a stochastic analysis viewcited by 1
  5. 2024Convergence rates of stochastic gradient method with independent sequences of step-size and momentum weightcited by 0
  6. 2025GradES: Significantly Faster Training in Transformers with Gradient-Based Early Stoppingcited by 0
  7. 2026Optimal Asymptotic Rates for (Stochastic) Gradient Descent under the Local PL-Condition: A Geometric Approachcited by 0