mlsanityData to 14 Oct 2026

Stochastic Gradient Descent Revisited

arXiv 2412.06070 ↗First posted 8 Dec 2024math.OC math.PR stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Stochastic gradient descent (SGD) has been a go-to algorithm for nonconvex stochastic optimization problems arising in machine learning. Its theory however often requires a strong framework to guarantee convergence properties. We hereby present a full scope convergence study of biased nonconvex SGD, including weak convergence, function-value convergence and global convergence, and also provide subsequent convergence rates and complexities, all under relatively mild conditions in comparison with literature.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 50 references that are in the map. First posted on arXiv 8 Dec 2024.

Builds on

24 of 50 references.

References · 24

  1. 2021Convex optimizationcited by 2,515
  2. 2016An overview of gradient descent optimization algorithmscited by 1,033
  3. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  4. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  5. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  6. 2020Better Theory for SGD in the Nonconvex Worldcited by 158
  7. 2018Stochastic subgradient method converges on tame functionscited by 124
  8. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 80
  9. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
  10. 2023A Guide Through the Zoo of Biased SGDcited by 40
  11. 2023Revisiting the Last-Iterate Convergence of Stochastic Gradient Methodscited by 38
  12. 2020High Probability Convergence Bounds for Non-convex Stochastic Gradient Descent with Sub-Weibull Noisecited by 31
12 more
  1. 2019Parameter-Free Online Convex Optimization with Sub-Exponential Noisecited by 28
  2. 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 28
  3. 2014Splitting methods with variable metric for KL functionscited by 26
  4. 2022Convergence of gradient descent for deep neural networkscited by 24
  5. 2023Aiming towards the minimizers: fast convergence of SGD for overparametrized problemscited by 22
  6. 2014Global convergence of splitting methods for nonconvex composite optimizationcited by 19
  7. 2021Global Convergence and Stability of Stochastic Gradient Descentcited by 17
  8. 2020On the Impossibility of Global Convergence in Multi-Loss Optimizationcited by 16
  9. 2024Almost sure convergence rates of stochastic gradient methods under gradient dominationcited by 5
  10. 2024Convergence of SGD with momentum in the nonconvex case: A time window-based analysiscited by 4
  11. 2015A Stochastic Majorize-Minimize Subspace Algorithm for Online Penalized Least Squares Estimationcited by 3
  12. 2023Convergence of stochastic gradient descent under a local Lojasiewicz condition for deep neural networkscited by 3

Built on by

Not cited yet.