mlsanityData to 7 Oct 2026

An Improved Analysis of Stochastic Gradient Descent with Momentum

arXiv 2007.07989 ↗First posted 15 Jul 2020math.OCcited by 117

Not on the map. Why?

Abstract (arXiv)

SGD with momentum (SGDM) has been widely applied in many machine learning tasks, and it is often applied with dynamic stepsizes and momentum weights tuned in a stagewise manner. Despite of its empirical advantage over SGD, the role of momentum is still unclear in general since previous analyses on SGDM either provide worse convergence bounds than those of SGD, or assume Lipschitz or quadratic objectives, which fail to hold in practice. Furthermore, the role of dynamic parameters has not been addressed. In this work, we show that SGDM converges as fast as SGD for smooth objectives under both strongly convex and nonconvex settings. We also establish textitthe first convergence guarantee for the multistage setting, and show that the multistage strategy is beneficial for SGDM compared to using fixed parameters. Finally, we verify these theoretical claims by numerical experiments.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 12; 2023: 20; 2024: 31; 2025: 30; 2026: 15.

117 papers.

Sort:

By area:

  1. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  2. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 100
  3. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  4. 2021DecentLaM: Decentralized Momentum SGD for Large-batch Deep Trainingcited by 40
  5. 2022High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsizecited by 40
  6. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  7. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  8. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 37
  9. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
  10. 2023Momentum Provably Improves Error Feedback!cited by 31
  11. 2022Provable Adaptivity of Adam under Non-uniform Smoothnesscited by 30
  12. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 21
38 more
  1. 2022Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Datacited by 20
  2. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 17
  3. 2025MUR: Momentum Uncertainty guided Reasoning for Large Language Modelscited by 17
  4. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 16
  5. 2023On the Role of Server Momentum in Federated Learningcited by 16
  6. 2024Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variancecited by 14
  7. 2021Unified Convergence Analysis for Adaptive Optimization with Moving Average Estimatorcited by 13
  8. 2024Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performancecited by 13
  9. 2022On the fast convergence of minibatch heavy ball momentumcited by 11
  10. 2022Provable Acceleration of Heavy Ball beyond Quadratics for a Class of Polyak-Lojasiewicz Functions when the Non-Convexity is Averaged-Outcited by 11
  11. 2025Refining Adaptive Zeroth-Order Optimization at Easecited by 11
  12. 2021Minibatch and Momentum Model-based Methods for Stochastic Weakly Convex Optimizationcited by 10
  13. 2021On the Last Iterate Convergence of Momentum Methodscited by 9
  14. 2021On the Hyperparameters in Stochastic Gradient Descent with Momentumcited by 9
  15. 2023Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noisecited by 9
  16. 2024Non-convex Stochastic Composite Optimization with Polyak Momentumcited by 9
  17. 2024Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networkscited by 9
  18. 2024A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGDcited by 9
  19. 2024On the Performance Analysis of Momentum Method: A Frequency Domain Perspectivecited by 8
  20. 2025Memory-Efficient LLM Pretraining via Minimalist Optimizer Designcited by 8
  21. 2026Momentum Further Constrains Sharpness at the Edge of Stochastic Stabilitycited by 8
  22. 2022On the Convergence of mSGD and AdaGrad for Stochastic Optimizationcited by 7
  23. 2024Fusion Transformer with Object Mask Guidance for Image Forgery Analysiscited by 7
  24. 2024FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Trainingcited by 7
  25. 2026Deriving Hyperparameter Scaling Laws via Modern Optimization Theorycited by 7
  26. 2021EF21 with Bells & Whistles: Six Algorithmic Extensions of Modern Error Feedbackcited by 6
  27. 2023Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descentcited by 6
  28. 2023Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normalitycited by 6
  29. 2024Towards Exact Gradient-based Training on Analog In-memory Computingcited by 6
  30. 2025Convergence Analysis of the Lion Optimizer in Centralized and Distributed Settingscited by 6
  31. 2024Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance Reductioncited by 5
  32. 2025Scaling Textual Gradients via Sampling-Based Momentumcited by 5
  33. 2023Fast Dual Subgradient Optimization of the Integrated Transportation Distance Between Stochastic Kernelscited by 4
  34. 2024Sobolev Training for Operator Learningcited by 4
  35. 2025Simple Convergence Proof of Adam From a Sign-like Descent Perspectivecited by 4
  36. 2023Deep Unrolling Networks with Recurrent Momentum Acceleration for Nonlinear Inverse Problemscited by 3
  37. 2023Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentumcited by 3
  38. 2024On the Convergence of FedProx with Extrapolation and Inexact Proxcited by 3

50 most cited of 117 All 117 (JSON)