# Stochastic optimization with momentum: convergence, fluctuations, and traps avoidance

arXiv: https://arxiv.org/abs/2012.04002 · HTML: https://mlsanity.com/paper/2012.04002/

## Key facts

- Cited by 10 papers in the map (data through 13 Oct 2026).
- First posted on arXiv 7 Dec 2020.

## Built on by (most cited first, 10 of 10)

- [Sharp higher order convergence rates for the Adam optimizer](https://mlsanity.com/paper/2504.19426/) (2025)
- [Efficiency Ordering of Stochastic Gradient Descent](https://mlsanity.com/paper/2209.07446/) (2022)
- [High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariance](https://mlsanity.com/paper/2304.00707/) (2023)
- [On the Algorithmic Stability and Generalization of Adaptive Optimization Methods](https://mlsanity.com/paper/2211.03970/) (2022)
- [Stable Anderson Acceleration for Deep Learning](https://mlsanity.com/paper/2110.14813/) (2021)
- [Anderson Acceleration For Bioinformatics-Based Machine Learning](https://mlsanity.com/paper/2302.00347/) (2023)
- [Accelerating Distributed Stochastic Optimization via Self-Repellent Random Walks](https://mlsanity.com/paper/2401.09665/) (2024)
- [Stochastic Subgradient Descent on a Generic Definable Function Converges to a Minimizer](https://mlsanity.com/paper/2109.02455/) (2021)
- [Statistical Inference for Linear Functionals of Online SGD in High-dimensional Linear Regression](https://mlsanity.com/paper/2302.09727/) (2023)
- [Statistical Inference for Linear Functionals of Online Least-squares SGD when t gtrsim d1+δ](https://mlsanity.com/paper/2510.19734/) (2025)

## Abstract (arXiv)

In this paper, a general stochastic optimization procedure is studied, unifying several variants of the stochastic gradient descent such as, among others, the stochastic heavy ball method, the Stochastic Nesterov Accelerated Gradient algorithm (S-NAG), and the widely used Adam algorithm. The algorithm is seen as a noisy Euler discretization of a non-autonomous ordinary differential equation, recently introduced by Belotto da Silva and Gazeau, which is analyzed in depth. Assuming that the objective function is non-convex and differentiable, the stability and the almost sure convergence of the iterates to the set of critical points are established. A noteworthy special case is the convergence proof of S-NAG in a non-convex setting. Under some assumptions, the convergence rate is provided under the form of a Central Limit Theorem. Finally, the non-convergence of the algorithm to undesired critical points, such as local maxima or saddle points, is established. Here, the main ingredient is a new avoidance of traps result for non-autonomous settings, which is of independent interest.