mlsanityData to 7 Oct 2026

Global Convergence and Stability of Stochastic Gradient Descent

arXiv 2110.01663 ↗First posted 4 Oct 2021cs.LG math.OCcited by 17

Not on the map. Why?

Abstract (arXiv)

In machine learning, stochastic gradient descent (SGD) is widely deployed to train models using highly non-convex objectives with equally complex noise models. Unfortunately, SGD theory often makes restrictive assumptions that fail to capture the non-convexity of real problems, and almost entirely ignore the complex noise models that exist in practice. In this work, we make substantial progress on this shortcoming. First, we establish that SGD's iterates will either globally converge to a stationary point or diverge under nearly arbitrary nonconvexity and noise models. Under a slightly more restrictive assumption on the joint behavior of the non-convexity and noise model that generalizes current assumptions in the literature, we show that the objective function cannot diverge, even if the iterates diverge. As a consequence of our results, SGD can be applied to a greater range of stochastic optimization problems with confidence about its global convergence behavior and stability.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 4 Oct 2021.

Builds on

23 of 37 references.

References · 23

  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  2. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  3. 2020Better Theory for SGD in the Nonconvex Worldcited by 156
  4. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  5. 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 143
  6. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
  7. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 79
  8. 2016Gradient Descent Learns Linear Dynamical Systemscited by 78
  9. 2020SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolationcited by 66
  10. 2018Stochastic Nested Variance Reduction for Nonconvex Optimizationcited by 63
  11. 2018Stochastic (Approximate) Proximal Point Methods: Convergence, Optimality, and Adaptivitycited by 61
  12. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
11 more
  1. 2019Sharp Analysis for Nonconvex SGD Escaping from Saddle Pointscited by 40
  2. 2021Convergence Rates of Stochastic Gradient Descent under Infinite Noise Variancecited by 34
  3. 2018On exponential convergence of SGD in non-convex over-parametrized learningcited by 33
  4. 2020Convergence of constant step stochastic gradient descent for non-smooth non-convex functionscited by 27
  5. 2018Analysis of nonsmooth stochastic approximation: the differential inclusion approachcited by 21
  6. 2020Stopping Criteria for, and Strong Convergence of, Stochastic Gradient Descent on Bottou-Curtis-Nocedal Functionscited by 13
  7. 2021Stochastic Gradient Descent on Nonconvex Functions with General Noise Modelscited by 7
  8. 2017Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigmscited by 5
  9. 2019A Stochastic Gradient Method with Biased Estimation for Faster Nonconvex Optimizationcited by 3
  10. 2017Diminishing Batch Normalizationcited by 1
  11. 2020Stochastic Approximation for High-frequency Observations in Data Assimilationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 3; 2025: 8; 2026: 2.

17 papers.

Sort:

  1. 2023Beyond Uniform Smoothness: A Stopped Analysis of Adaptive SGDcited by 41
  2. 2023Distributed Stochastic Optimization under a General Variance Conditioncited by 4
  3. 2025Theoretical Analysis on how Learning Rate Warmup Accelerates Convergencecited by 4
  4. 2025Statistical Inference for Online Algorithmscited by 2
  5. 2022Zeroth-Order Actor-Critic: An Evolutionary Framework for Sequential Decision Problemscited by 1
  6. 2025HRP: High-Rank Preheating for Superior LoRA Initializationcited by 1
  7. 2025ADAM Optimization with Adaptive Batch Selectioncited by 1
  8. 2026Beyond Student: An Asymmetric Network for Neural Network Inheritancecited by 1
  9. 2026Tuning the burn-in phase in training recurrent neural networks improves their performancecited by 1
  10. 2022A Local Convergence Theory for the Stochastic Gradient Descent Method in Non-Convex Optimization With Non-isolated Local Minimacited by 0
  11. 2024Dynamic Decoupling of Placid Terminal Attractor-based Gradient Descent Algorithmcited by 0
  12. 2024Recent Advances in Non-convex Smoothness Conditions and Applicability to Deep Linear Neural Networkscited by 0
5 more
  1. 2024Stochastic Gradient Descent Revisitedcited by 0
  2. 2025Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networkscited by 0
  3. 2025First-ish Order Methods: Hessian-aware Scalings of Gradient Descentcited by 0
  4. 2025AI-based modular warning machine for risk identification in proximity healthcarecited by 0
  5. 2025Convergence of a class of gradient-free optimisation schemes when the objective function is noisy, irregular, or bothcited by 0