mlsanityData to 14 Oct 2026

A Unified Convergence Theorem for Stochastic Optimization Methods

arXiv 2206.03907 ↗First posted 8 Jun 2022math.OC cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

In this work, we provide a fundamental unified convergence theorem used for deriving expected and almost sure convergence results for a series of stochastic optimization methods. Our unified theorem only requires to verify several representative conditions and is not tailored to any specific algorithm. As a direct application, we recover expected and almost sure convergence results of the stochastic gradient method (SGD) and random reshuffling (RR) under more general settings. Moreover, we establish new expected and almost sure convergence results for the stochastic proximal gradient method (prox-SGD) and stochastic model-based methods (SMM) for nonsmooth nonconvex optimization problems. These applications reveal that our unified theorem provides a plugin-type convergence analysis and strong convergence guarantees for a wide class of stochastic optimization methods.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 22 of its 55 references that are in the map. First posted on arXiv 8 Jun 2022.

Builds on

22 of 55 references.

References · 22

  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  2. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  3. 2018Stochastic model-based minimization of weakly convex functionscited by 159
  4. 2020Better Theory for SGD in the Nonconvex Worldcited by 158
  5. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
  6. 2013Mini-batch Stochastic Approximation Methods for Nonconvex Stochastic Composite Optimizationcited by 143
  7. 2018Stochastic subgradient method converges on tame functionscited by 124
  8. 2020Random Reshuffling: Simple Analysis with Vast Improvementscited by 115
  9. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 80
  10. 2015Why Random Reshuffling Beats Stochastic Gradient Descentcited by 79
  11. 2020SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolationcited by 67
  12. 2016Error bounds, quadratic growth, and linear convergence of proximal methodscited by 60
10 more
  1. 2020A Unified Convergence Analysis for Shuffling-Type Gradient Methodscited by 58
  2. 2020The limits of min-max optimization algorithms: convergence to spurious non-critical setscited by 50
  3. 2017Stochastic Methods for Composite and Weakly Convex Optimization Problemscited by 48
  4. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
  5. 2018Analysis of nonsmooth stochastic approximation: the differential inclusion approachcited by 23
  6. 2014Convergence of Stochastic Proximal Gradient Algorithmcited by 17
  7. 2021Convergence of Random Reshuffling Under The Kurdyka-Lojasiewicz Inequalitycited by 16
  8. 2020Variable Smoothing for Weakly Convex Composite Functionscited by 12
  9. 2022A Semismooth Newton Stochastic Proximal Point Algorithm with Variance Reductioncited by 5
  10. 2020Stochastic Proximal Gradient Methods for Nonconvex Problems in Hilbert Spacescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 2; 2025: 3; 2026: 1.

12 papers.

  1. 2023Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Casecited by 14
  2. 2023Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inferencecited by 6
  3. 2024FiRe: Fixed-points of Restoration Priors for Solving Inverse Problemscited by 6
  4. 2023A Normal Map-Based Proximal Stochastic Gradient Method: Convergence and Identification Propertiescited by 5
  5. 2023High Probability Guarantees for Random Reshufflingcited by 5
  6. 2023Distributed Stochastic Optimization under a General Variance Conditioncited by 4
  7. 2023A New Random Reshuffling Method for Nonsmooth Nonconvex Finite-sum Optimizationcited by 4
  8. 2024A quantitative Robbins-Siegmund theoremcited by 4
  9. 2025An abstract effective convergence theorem for stochastic processes, with applications to stochastic approximationcited by 1
  10. 2025Quantitative Convergence Analysis of Projected Stochastic Gradient Descent for Non-Convex Losses via the Goldstein Subdifferentialcited by 0
  11. 2025Towards Continuous-Time Approximations for Stochastic Gradient Descent without Replacementcited by 0
  12. 2026Stochastic Saddle Avoidance Beyond Unit Excitation and Smoothness: A Pathwise Lyapunov-Perron Frameworkcited by 0