# On the Convergence of Adam, Revisited

arXiv: https://arxiv.org/abs/2607.03519 · HTML: https://mlsanity.com/paper/2607.03519/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 25 references that are in the map.
- First posted on arXiv 3 Jul 2026.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Adaptive Gradient Methods with Dynamic Bound of Learning Rate](https://mlsanity.com/paper/1902.09843/) (2019)
- [Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models](https://mlsanity.com/paper/2208.06677/) (2022)
- [Adam Can Converge Without Any Modification On Update Rules](https://mlsanity.com/paper/2208.09632/) (2022)
- [A new regret analysis for Adam-type algorithms](https://mlsanity.com/paper/2003.09729/) (2020)
- [A general system of differential equations to model first order adaptive algorithms](https://mlsanity.com/paper/1810.13108/) (2018)
- [Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise](https://mlsanity.com/paper/2402.01567/) (2024)
- [Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate](https://mlsanity.com/paper/1805.07557/) (2018)
- [Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates](https://mlsanity.com/paper/2407.08100/) (2024)
- [Non-Convergence and Limit Cycles in the Adam optimizer](https://mlsanity.com/paper/2210.02070/) (2022)
- [Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks](https://mlsanity.com/paper/2402.05155/) (2024)
- [Muon Does Not Converge on Convex Lipschitz Functions](https://mlsanity.com/paper/2605.08980/) (2026)
- [Divergence Results and Convergence of a Variance Reduced Version of ADAM](https://mlsanity.com/paper/2210.05607/) (2022)
- [Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activation](https://mlsanity.com/paper/2410.10533/) (2024)
- [Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networks](https://mlsanity.com/paper/2503.01660/) (2025)
- [Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer](https://mlsanity.com/paper/2511.06675/) (2025)
- [Towards Understanding Adam Convergence on Highly Degenerate Polynomials](https://mlsanity.com/paper/2603.09581/) (2026)
- [Methods of Nonconvex Optimization](https://mlsanity.com/paper/2406.10406/) (2024)
- [Divergence of the ADAM algorithm with fixed-stepsize: a (very) simple example](https://mlsanity.com/paper/2308.00720/) (2023)
- [Adam Converges Without Any Modification On Update Rules](https://mlsanity.com/paper/2603.02092/) (2026)

## Built on by (most cited first, 1 of 1)

- [Convergence rates for the RMSprop optimizer with full control of the hyperparameters](https://mlsanity.com/paper/2608.30382/) (2026)

## Abstract (arXiv)

We show that projected Adam for online optimization with arbitrary moment decay parameters β1,β2∈[0,1) can have average regret bounded away from zero. A similar result of Reddi-Kale-Kumar from 2018 required β1<√β2. Similar to their result, we use a three-periodic sequence of linear functions on [-1,1] with slopes c,-1,-1, though we use c slightly larger than 2. This nonzero average regret result extends to Adam variants such as AdamW, RMSProp, NAdam, Adan, AdaMax, Muon, and to an i.i.d. variant of the three-periodic sequence of slopes for Adam.