# A Lyapunov Analysis of Momentum Methods in Optimization

arXiv: https://arxiv.org/abs/1611.02635 · HTML: https://mlsanity.com/paper/1611.02635/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2016.

## Built on by (most cited first, 28 of 28)

- [Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks](https://mlsanity.com/paper/2105.02716/) (2021)
- [Cautious Optimizers: Improving Training with One Line of Code](https://mlsanity.com/paper/2411.16085/) (2024)
- [LyaNet: A Lyapunov Framework for Training Neural ODEs](https://mlsanity.com/paper/2202.02526/) (2022)
- [Heavy Ball Neural Ordinary Differential Equations](https://mlsanity.com/paper/2110.04840/) (2021)
- [On Almost Sure Convergence Rates of Stochastic Gradient Methods](https://mlsanity.com/paper/2202.04295/) (2022)
- [Continuous vs. Discrete Optimization of Deep Neural Networks](https://mlsanity.com/paper/2107.06608/) (2021)
- [A Continuized View on Nesterov Acceleration for Stochastic Gradient Descent and Randomized Gossip](https://mlsanity.com/paper/2106.07644/) (2021)
- [Continuous-Time Meta-Learning with Forward Mode Differentiation](https://mlsanity.com/paper/2203.01443/) (2022)
- [Momentum Particle Maximum Likelihood](https://mlsanity.com/paper/2312.07335/) (2023)
- [Accelerating optimization over the space of probability measures](https://mlsanity.com/paper/2310.04006/) (2023)
- [Analyzing Human Models that Adapt Online](https://mlsanity.com/paper/2103.05746/) (2021)
- [Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization](https://mlsanity.com/paper/2102.11537/) (2021)
- [Potential Function-based Framework for Making the Gradients Small in Convex and Min-Max Optimization](https://mlsanity.com/paper/2101.12101/) (2021)
- [Grams: Gradient Descent with Adaptive Momentum Scaling](https://mlsanity.com/paper/2412.17107/) (2024)
- [A Contraction Theory Approach to Optimization Algorithms from Acceleration Flows](https://mlsanity.com/paper/2105.08832/) (2021)
- [Mean-field underdamped Langevin dynamics and its spacetime discretization](https://mlsanity.com/paper/2312.16360/) (2023)
- [Resonance in Weight Space: Covariate Shift Can Drive Divergence of SGD with Momentum](https://mlsanity.com/paper/2203.11992/) (2022)
- [Competitive Gradient Optimization](https://mlsanity.com/paper/2205.14232/) (2022)
- [A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networks](https://mlsanity.com/paper/2206.02001/) (2022)
- [Beyond first-order methods for non-convex non-concave min-max optimization](https://mlsanity.com/paper/2304.08389/) (2023)
- [On the Generalization of Stochastic Gradient Descent with Momentum](https://mlsanity.com/paper/2102.13653/) (2021)
- [A High-order Tuner for Accelerated Learning and Control](https://mlsanity.com/paper/2103.12868/) (2021)
- [Momentum-inspired Low-Rank Coordinate Descent for Diagonally Constrained SDPs](https://mlsanity.com/paper/2106.08775/) (2021)
- [Accelerate Distributed Stochastic Descent for Nonconvex Optimization with Momentum](https://mlsanity.com/paper/2110.00625/) (2021)
- [A More Stable Accelerated Gradient Method Inspired by Continuous-Time Perspective](https://mlsanity.com/paper/2112.04922/) (2021)
- [Accelerated Gradient Flow: Risk, Stability, and Implicit Regularization](https://mlsanity.com/paper/2201.08311/) (2022)
- [A General Continuous-Time Formulation of Stochastic ADMM and Its Variants](https://mlsanity.com/paper/2404.14358/) (2024)
- [Heavy-Ball Momentum Method in Continuous Time and Discretization Error Analysis](https://mlsanity.com/paper/2506.14806/) (2025)

## Abstract (arXiv)

Momentum methods play a significant role in optimization. Examples include Nesterov's accelerated gradient method and the conditional gradient algorithm. Several momentum methods are provably optimal under standard oracle models, and all use a technique called estimate sequences to analyze their convergence properties. The technique of estimate sequences has long been considered difficult to understand, leading many researchers to generate alternative, "more intuitive" methods and analyses. We show there is an equivalence between the technique of estimate sequences and a family of Lyapunov functions in both continuous and discrete time. This connection allows us to develop a simple and unified analysis of many existing momentum algorithms, introduce several new algorithms, and strengthen the connection between algorithms and continuous-time dynamical systems.