mlsanityData to 14 Oct 2026

Understanding the Role of Momentum in Stochastic Gradient Methods

arXiv 1910.13962 ↗First posted 30 Oct 2019cs.LG math.OC stat.MLcited by 44

Not on the map. Why?

Abstract (arXiv)

The use of momentum in stochastic gradient methods has become a widespread practice in machine learning. Different variants of momentum, including heavy-ball momentum, Nesterov's accelerated gradient (NAG), and quasi-hyperbolic momentum (QHM), have demonstrated success on various tasks. Despite these empirical successes, there is a lack of clear understanding of how the momentum parameters affect convergence and various performance measures of different algorithms. In this paper, we use the general formulation of QHM to give a unified analysis of several popular algorithms, covering their asymptotic convergence conditions, stability regions, and properties of their stationary distributions. In addition, by combining the results on convergence rates and stationary distributions, we obtain sometimes counter-intuitive practical guidelines for setting the learning rate and momentum parameters.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 12; 2022: 4; 2023: 4; 2024: 8; 2025: 7; 2026: 9.

44 papers.

Sort:

  1. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  2. 2021DecentLaM: Decentralized Momentum SGD for Large-batch Deep Trainingcited by 40
  3. 2021Strength of Minibatch Noise in SGDcited by 39
  4. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 18
  5. 2022On the fast convergence of minibatch heavy ball momentumcited by 13
  6. 2024Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performancecited by 13
  7. 2021Minibatch and Momentum Model-based Methods for Stochastic Weakly Convex Optimizationcited by 10
  8. 2024Information-Theoretic Generalization Bounds for Deep Neural Networkscited by 10
  9. 2026Momentum Further Constrains Sharpness at the Edge of Stochastic Stabilitycited by 10
  10. 2023Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normalitycited by 9
  11. 2023Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noisecited by 9
  12. 2021Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimizationcited by 8
32 more
  1. 2021The Role of Momentum Parameters in the Optimal Convergence of Adaptive Polyak's Heavy-ball Methodscited by 7
  2. 2022On the Convergence of mSGD and AdaGrad for Stochastic Optimizationcited by 7
  3. 2021Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimizationcited by 6
  4. 2021Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problemscited by 5
  5. 2023Enhance Diffusion to Improve Robust Generalizationcited by 3
  6. 2021Bandwidth-based Step-Sizes for Non-Convex Stochastic Optimizationcited by 2
  7. 2021Gradient Temporal Difference with Momentum: Stability and Convergencecited by 2
  8. 2024Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guaranteescited by 2
  9. 2024Explicit and Implicit Graduated Optimization in Deep Neural Networkscited by 2
  10. 2025Increasing Batch Size Improves Convergence of Stochastic Gradient Descent with Momentumcited by 2
  11. 2025AdamNX: An Adam improvement algorithm based on a novel exponential decay mechanism for the second-order moment estimatecited by 2
  12. 2021Scaling transition from momentum stochastic gradient descent to plain stochastic gradient descentcited by 1
  13. 2022Computing the Variance of Shuffling Stochastic Gradient Algorithms via Power Spectral Density Analysiscited by 1
  14. 2022Tradeoffs between convergence rate and noise amplification for momentum-based accelerated optimization algorithmscited by 1
  15. 2024Effectively Leveraging Momentum Terms in Stochastic Line Search Frameworks for Fast Optimization of Finite-Sum Problemscited by 1
  16. 2024Sharpness-Aware Minimization with Adaptive Regularization for Training Deep Neural Networkscited by 1
  17. 2025Both Asymptotic and Non-Asymptotic Convergence of Quasi-Hyperbolic Momentum using Increasing Batch Sizecited by 1
  18. 2025Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysiscited by 1
  19. 2025Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimizationcited by 1
  20. 2026Stochastic Gradient Descent with Momentum is Algorithmically Stablecited by 1
  21. 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1
  22. 2021Scale Invariant Monte Carlo under Linear Function Approximation with Curvature based step-sizecited by 0
  23. 2024A Selective Review on Statistical Methods for Massive Data Computation: Distributed Computing, Subsampling, and Minibatch Techniquescited by 0
  24. 2024SOREL: A Stochastic Algorithm for Spectral Risks Minimizationcited by 0
  25. 2025Stochastic Gradient Descent in Non-Convex Problems: Asymptotic Convergence with Relaxed Step-Size via Stopping Time Methodscited by 0
  26. 2025Why Federated Optimization Fails to Achieve Perfect Fitting? A Theoretical Perspective on Client-Side Optimacited by 0
  27. 2026Ada2MS: A Hybrid Optimization Algorithm Based on Exponential Mixing of Elementwise and Global Second-Moment Estimatescited by 0
  28. 2026Dynamics of Stochastic Momentum with Sparse Updates in High Dimensionscited by 0
  29. 2026VCMM: Variance-Calibrated Momentum for Multimodal Learningcited by 0
  30. 2026A Journey to the Edge of Stabilitycited by 0
  31. 2026Stochastic Heavy Ball with Polyak Step Size and Armijo Line Search: A General Convergence Analysiscited by 0
  32. 2026A Nonlinear Time-Series Theory for Constant-step SGD with Momentumcited by 0