Convergence of Momentum-Based Optimization Algorithms with Time-Varying Parameters
Not on the map. Why?
Abstract (arXiv)
In this paper, we present a unified algorithm for stochastic optimization that makes use of a "momentum" term; in other words, the stochastic gradient depends not only on the current true gradient of the objective function, but also on the true gradient at the previous iteration. Our formulation includes the Stochastic Heavy Ball (SHB) and the Stochastic Nesterov Accelerated Gradient (SNAG) algorithms as special cases. In addition, in our formulation, the momentum term is allowed to vary as a function of time (i.e., the iteration counter). The assumptions on the stochastic gradient are the most general in the literature, in that it can be biased, and have a conditional variance that grows in an unbounded fashion as a function of time. This last feature is crucial in order to make the theory applicable to "zero-order" methods, where the gradient is estimated using just two function evaluations. We present a set of sufficient conditions for the convergence of the unified algorithm. These conditions are natural generalizations of the familiar Robbins-Monro and Kiefer-Wolfowitz-Blum conditions for standard stochastic gradient descent. We also analyze another method from the literature for the SHB algorithm with a time-varying momentum parameter, and show that it is impracticable.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 15 of its 36 references that are in the map. First posted on arXiv 13 Jun 2025.
Builds on
15 of 36 references.
References · 15
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
- 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
- 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
- 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 206
- 2020Better Theory for SGD in the Nonconvex Worldcited by 158
- 2014Global convergence of the Heavy-ball method for convex optimizationcited by 61
- 2018A Unified Analysis of Stochastic Momentum Methods for Deep Learningcited by 55
- 2016Stochastic Heavy Ballcited by 42
- 2022On Almost Sure Convergence Rates of Stochastic Gradient Methodscited by 35
- 2018A Unified Analysis of AdaGrad with Weighted Aggregation and Momentum Accelerationcited by 15
- 2023Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applicationscited by 15
- 2021Convergence rates for the Heavy-Ball continuous dynamics for non-convex optimization, under Polyak-L ojasiewicz conditioncited by 11
3 more
- 2021Convergence of sequences: a surveycited by 10
- 2018Optimal convergence rates for Nesterov accelerationcited by 8
- 2022Last-iterate convergence analysis of stochastic momentum methods for neural networkscited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.