mlsanityData to 14 Oct 2026

A Lyapunov Analysis of Momentum Methods in Optimization

arXiv 1611.02635 ↗First posted 8 Nov 2016math.OC cs.DScited by 28

Not on the map. Why?

Abstract (arXiv)

Momentum methods play a significant role in optimization. Examples include Nesterov's accelerated gradient method and the conditional gradient algorithm. Several momentum methods are provably optimal under standard oracle models, and all use a technique called estimate sequences to analyze their convergence properties. The technique of estimate sequences has long been considered difficult to understand, leading many researchers to generate alternative, "more intuitive" methods and analyses. We show there is an equivalence between the technique of estimate sequences and a family of Lyapunov functions in both continuous and discrete time. This connection allows us to develop a simple and unified analysis of many existing momentum algorithms, introduce several new algorithms, and strengthen the connection between algorithms and continuous-time dynamical systems.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 13; 2022: 7; 2023: 4; 2024: 3; 2025: 1.

28 papers.

Sort:

  1. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  2. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 47
  3. 2022LyaNet: A Lyapunov Framework for Training Neural ODEscited by 44
  4. 2021Heavy Ball Neural Ordinary Differential Equationscited by 42
  5. 2022On Almost Sure Convergence Rates of Stochastic Gradient Methodscited by 35
  6. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  7. 2021A Continuized View on Nesterov Acceleration for Stochastic Gradient Descent and Randomized Gossipcited by 15
  8. 2022Continuous-Time Meta-Learning with Forward Mode Differentiationcited by 12
  9. 2023Momentum Particle Maximum Likelihoodcited by 12
  10. 2023Accelerating optimization over the space of probability measurescited by 10
  11. 2021Analyzing Human Models that Adapt Onlinecited by 8
  12. 2021Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimizationcited by 6
16 more
  1. 2021Potential Function-based Framework for Making the Gradients Small in Convex and Min-Max Optimizationcited by 4
  2. 2024Grams: Gradient Descent with Adaptive Momentum Scalingcited by 4
  3. 2021A Contraction Theory Approach to Optimization Algorithms from Acceleration Flowscited by 3
  4. 2023Mean-field underdamped Langevin dynamics and its spacetime discretizationcited by 2
  5. 2022Resonance in Weight Space: Covariate Shift Can Drive Divergence of SGD with Momentumcited by 1
  6. 2022Competitive Gradient Optimizationcited by 1
  7. 2022A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networkscited by 1
  8. 2023Beyond first-order methods for non-convex non-concave min-max optimizationcited by 1
  9. 2021On the Generalization of Stochastic Gradient Descent with Momentumcited by 0
  10. 2021A High-order Tuner for Accelerated Learning and Controlcited by 0
  11. 2021Momentum-inspired Low-Rank Coordinate Descent for Diagonally Constrained SDPscited by 0
  12. 2021Accelerate Distributed Stochastic Descent for Nonconvex Optimization with Momentumcited by 0
  13. 2021A More Stable Accelerated Gradient Method Inspired by Continuous-Time Perspectivecited by 0
  14. 2022Accelerated Gradient Flow: Risk, Stability, and Implicit Regularizationcited by 0
  15. 2024A General Continuous-Time Formulation of Stochastic ADMM and Its Variantscited by 0
  16. 2025Heavy-Ball Momentum Method in Continuous Time and Discretization Error Analysiscited by 0