mlsanityData to 7 Oct 2026

Momentum via Primal Averaging: Theoretical Insights and Learning Rate Schedules for Non-Convex Optimization

arXiv 2010.00406 ↗First posted 1 Oct 2020cs.LG math.OC stat.MLcited by 19

Not on the map. Why?

Abstract (arXiv)

Momentum methods are now used pervasively within the machine learning community for training non-convex models such as deep neural networks. Empirically, they out perform traditional stochastic gradient descent (SGD) approaches. In this work we develop a Lyapunov analysis of SGD with momentum (SGD+M), by utilizing a equivalent rewriting of the method known as the stochastic primal averaging (SPA) form. This analysis is much tighter than previous theory in the non-convex case, and due to this we are able to give precise insights into when SGD+M may out-perform SGD, and what hyper-parameter schedules will work and why.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 5; 2024: 2; 2025: 6; 2026: 1.

19 papers.

Sort:

By area:

  1. 2024The Road Less Scheduledcited by 158
  2. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  3. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  4. 2022High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsizecited by 40
  5. 2023Momentum Provably Improves Error Feedback!cited by 31
  6. 2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
  7. 2022Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Datacited by 20
  8. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 17
  9. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 16
  10. 2022Low-Variance Forward Gradients using Direct Feedback Alignment and Momentumcited by 7
  11. 2025SNOO: Step-K Nesterov Outer Optimizer - The Surprising Effectiveness of Nesterov Momentum Applied to Pseudo-Gradientscited by 6
  12. 2025Smoothing DiLoCo with Primal Averaging for Faster Training of LLMscited by 5
7 more
  1. 2025Understanding SGD with Exponential Moving Average: A Case Study in Linear Regressioncited by 1
  2. 2025Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysiscited by 1
  3. 2025Analysis of Schedule-Free Nonconvex Optimizationcited by 1
  4. 2025Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimizationcited by 1
  5. 2022Formal guarantees for heuristic optimization algorithms used in machine learningcited by 0
  6. 2024(Accelerated) Noise-adaptive Stochastic Heavy-Ball Momentumcited by 0
  7. 2026Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalizationcited by 0