Convergence rates for the RMSprop optimizer with full control of the hyperparameters
Not on the map. Why?
Abstract (arXiv)
Popular adaptive stochastic gradient descent (SGD) methods to train artificial intelligence (AI) systems include the RMSprop, the Adam, and the AdamW optimizers, where the adaptivity parts in Adam and AdamW basically just coincide with RMSprop. Such adaptive methods involve several hyperparameters including the regularization parameter ε (which ensures that one does not divide by 0 and is often chosen to be very close to zero such as 10-8 in PyTorch by default) and the second moment decay parameter β (which is often chosen to be very close to 1 such as 0.99 (RMSprop) and 0.999 (Adam and AdamW) in PyTorch by default). Despite the high relevance of such methods, it remains an open research problem to provide error estimates for such methods with the error constants being not exploding but uniformly bounded with the respect to the hyperparameters, even in the situation of convex stochastic optimization problems. It is the key contribution of this work to essentially solve this problem for RMSprop. Specifically, we bound the expectation of the stopped evaluation of the objective function at the RMSprop process from above by the sum of an initialization term that decays exponentially in the training time, a stochastic approximation remainder of order γn, and a memory error of order ( 1 - β)2 with the error constants being uniformly controlled over all admissible choices of the step sizes, the second moment decay parameter β and the regularization parameter ε∈[0,1] (also covering ε=0). Our non-asymptotic error estimates hold not just for all sufficiently large n but hold for every gradient step n=1,2,3,... with all error constants being explicitly specified. The key innovative new feature in the proof of our analysis are suitable inverse moment bounds for the second moment process in RMSprop.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 41 references that are in the map. First posted on arXiv 31 Aug 2026.
Builds on
34 of 41 references.
References · 34
- 2016An overview of gradient descent optimization algorithmscited by 1,033
- 2019On the Convergence of Adam and Beyondcited by 841
- 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
- 2020A Simple Convergence Proof of Adam and Adagradcited by 171
- 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
- 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
- 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
- 2023Convergence of Adam Under Relaxed Assumptionscited by 99
- 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
- 2017Variants of RMSProp and Adagrad with Logarithmic Regret Boundscited by 50
- 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 49
- 2018Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov accelerationcited by 39
22 more
- 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
- 2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 33
- 2024On Convergence of Adam for Stochastic Optimization under Relaxed Assumptionscited by 21
- 2023Mathematical Introduction to Deep Learning: Methods, Implementations, and Theorycited by 20
- 2020Asymptotic study of stochastic adaptive algorithm in non-convex landscapecited by 16
- 2024Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning ratescited by 15
- 2024Convergence rates for the Adam optimizercited by 15
- 2024Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variancecited by 14
- 2018Lower error bounds for the stochastic gradient descent optimization algorithm: Sharp convergence rates for slowly and fast decaying learning ratescited by 11
- 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
- 2024Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activationcited by 8
- 2025Sharp higher order convergence rates for the Adam optimizercited by 8
- 2023Non asymptotic analysis of Adaptive stochastic gradient algorithms and applicationscited by 7
- 2025Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networkscited by 6
- 2025Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizercited by 6
- 2025Asymptotic stability properties and a priori bounds for Adam and other gradient descent optimization methodscited by 4
- 2026Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization methodcited by 3
- 2024Convergence of the Iterates for Momentum and RMSProp for Local Smooth Functions: Adaptation is the Keycited by 2
- 2026Global Stability and Step Size Robustness of RMSPropcited by 1
- 2026Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimizationcited by 1
- 2026On the Convergence of Adam, Revisitedcited by 1
- 2026Unified convergence analysis for gradient descent optimization methods in the training of deep neural networkscited by 1
Built on by
Not cited yet.