On the Convergence of Adam, Revisited
Not on the map. Why?
Abstract (arXiv)
We show that projected Adam for online optimization with arbitrary moment decay parameters β1,β2∈[0,1) can have average regret bounded away from zero. A similar result of Reddi-Kale-Kumar from 2018 required β1<√β2. Similar to their result, we use a three-periodic sequence of linear functions on [-1,1] with slopes c,-1,-1, though we use c slightly larger than 2. This nonzero average regret result extends to Adam variants such as AdamW, RMSProp, NAdam, Adan, AdaMax, Muon, and to an i.i.d. variant of the three-periodic sequence of slopes for Adam.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 25 references that are in the map. First posted on arXiv 3 Jul 2026.
Builds on
22 of 25 references.
References · 22
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019On the Convergence of Adam and Beyondcited by 834
- 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 181
- 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
- 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
- 2020A new regret analysis for Adam-type algorithmscited by 35
- 2018A general system of differential equations to model first order adaptive algorithmscited by 28
- 2024Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguisecited by 23
- 2018Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning ratecited by 16
- 2024Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning ratescited by 14
- 2022Non-Convergence and Limit Cycles in the Adam optimizercited by 9
10 more
- 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
- 2026Muon Does Not Converge on Convex Lipschitz Functionscited by 9
- 2022Divergence Results and Convergence of a Variance Reduced Version of ADAMcited by 8
- 2024Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activationcited by 8
- 2025Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networkscited by 6
- 2025Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizercited by 5
- 2024Methods of Nonconvex Optimizationcited by 3
- 2026Towards Understanding Adam Convergence on Highly Degenerate Polynomialscited by 3
- 2023Divergence of the ADAM algorithm with fixed-stepsize: a (very) simple examplecited by 2
- 2026Adam Converges Without Any Modification On Update Rulescited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.