mlsanityData to 7 Oct 2026

On Uniform Boundedness Properties of SGD and its Momentum Variants

arXiv 2201.10245 ↗First posted 25 Jan 2022cs.LG math.OCcited by 2

Not on the map. Why?

Abstract (arXiv)

A theoretical, and potentially also practical, problem with stochastic gradient descent is that trajectories may escape to infinity. In this note, we investigate uniform boundedness properties of iterates and function values along the trajectories of the stochastic gradient descent algorithm and its important momentum variant. Under smoothness and R-dissipativity of the loss function, we show that broad families of step-sizes, including the widely used step-decay and cosine with (or without) restart step-sizes, result in uniformly bounded iterates and function values. Several important applications that satisfy these assumptions, including phase retrieval problems, Gaussian mixture models, and some neural network classifiers, are discussed in detail. We further extend the uniform boundedness of SGD and its momentum variant under the generalized dissipativity for the functions whose tails grow slower than quadratic functions. This includes some interesting applications, for example, Bayesian logistic regression and logistic regression with ℓ1 regularization.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 21 of its 36 references that are in the map. First posted on arXiv 25 Jan 2022.

Builds on

21 of 36 references.

References · 21

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  3. 2018Stochastic subgradient method converges on tame functionscited by 120
  4. 2014Guaranteed Matrix Completion via Non-convex Factorizationcited by 81
  5. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 79
  6. 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 67
  7. 2020On the Convergence of Langevin Monte Carlo: The Interplay between Tail Growth and Smoothnesscited by 55
  8. 2020An Analysis of Constant Step Size SGD in the Non-convex Regime: Asymptotic Normality and Biascited by 45
  9. 2018Global Non-convex Optimization with Discretized Diffusionscited by 44
  10. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 38
  11. 2019Online Stochastic Gradient Descent with Arbitrary Initialization Solves Non-smooth, Non-convex Phase Retrievalcited by 34
  12. 2020On Learning Rates and Schr"odinger Operatorscited by 33
9 more
  1. 2020A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performancecited by 26
  2. 2021On the Convergence of Step Decay Step-Size for Stochastic Optimizationcited by 21
  3. 2018Stochastic Optimization for DC Functions and Non-smooth Non-convex Regularizers with Non-asymptotic Convergencecited by 16
  4. 2021Stationary Behavior of Constant Stepsize SGD Type Algorithms: An Asymptotic Characterizationcited by 10
  5. 2021On the Convergence of Stochastic Gradient Descent with Bandwidth-based Step Sizecited by 9
  6. 2018Learning with Non-Convex Truncated Losses by SGDcited by 7
  7. 2021Stochastic Gradient Descent on Nonconvex Functions with General Noise Modelscited by 7
  8. 2019On the Convergence of (Stochastic) Gradient Descent with Extrapolation for Non-Convex Optimizationcited by 6
  9. 2021Bandwidth-based Step-Sizes for Non-Convex Stochastic Optimizationcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2.

2 papers.

  1. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  2. 2023Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normalitycited by 6