# On the Convergence Proof of AMSGrad and a New Version

arXiv: https://arxiv.org/abs/1904.03590 · HTML: https://mlsanity.com/paper/1904.03590/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2019.

## Built on by (most cited first, 21 of 21)

- [Toward Understanding Why Adam Converges Faster Than SGD for Transformers](https://mlsanity.com/paper/2306.00204/) (2023)
- [High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsize](https://mlsanity.com/paper/2204.02833/) (2022)
- [Order in the Court: Explainable AI Methods Prone to Disagreement](https://mlsanity.com/paper/2105.03287/) (2021)
- [SGD with AdaGrad Stepsizes: Full Adaptivity with High Probability to Unknown Parameters, Unbounded Gradients and Affine Variance](https://mlsanity.com/paper/2302.08783/) (2023)
- [A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing](https://mlsanity.com/paper/2205.04559/) (2022)
- [How Free is Parameter-Free Stochastic Optimization?](https://mlsanity.com/paper/2402.03126/) (2024)
- [Solving a class of non-convex min-max games using adaptive momentum methods](https://mlsanity.com/paper/2104.12676/) (2021)
- [A Decentralized Adaptive Momentum Method for Solving a Class of Min-Max Optimization Problems](https://mlsanity.com/paper/2106.06075/) (2021)
- [HesScale: Scalable Computation of Hessian Diagonals](https://mlsanity.com/paper/2210.11639/) (2022)
- [Expressive equivalence of classical and quantum restricted Boltzmann machines](https://mlsanity.com/paper/2502.17562/) (2025)
- [Adversarially Robust Kernel Smoothing](https://mlsanity.com/paper/2102.08474/) (2021)
- [DeMuon: A Decentralized Muon for Matrix Optimization over Graphs](https://mlsanity.com/paper/2510.01377/) (2025)
- [An Empirical Review of Optimization Techniques for Quantum Variational Circuits](https://mlsanity.com/paper/2202.01389/) (2022)
- [Investigation of Energy-efficient AI Model Architectures and Compression Techniques for "Green" Fetal Brain Segmentation](https://mlsanity.com/paper/2405.15778/) (2024)
- [ADAM Optimization with Adaptive Batch Selection](https://mlsanity.com/paper/2512.06795/) (2025)
- [Optimizing Convergence for Iterative Learning of ARIMA for Stationary Time Series](https://mlsanity.com/paper/2101.10037/) (2021)
- [Neural Optimizer Equation, Decay Function, and Learning Rate Schedule Joint Evolution](https://mlsanity.com/paper/2404.06679/) (2024)
- [Learning Rate Annealing Improves Tuning Robustness in Stochastic Optimization](https://mlsanity.com/paper/2503.09411/) (2025)
- [Images in Motion?: A First Look into Video Leakage in Collaborative Deep Learning](https://mlsanity.com/paper/2509.09742/) (2025)
- [Novel Dynamic Batch-Sensitive Adam Optimiser for Vehicular Accident Injury Severity Prediction](https://mlsanity.com/paper/2605.15083/) (2026)
- [A data-driven Fourier-mixture neural-network method for density estimation](https://mlsanity.com/paper/2605.18019/) (2026)

## Abstract (arXiv)

The adaptive moment estimation algorithm Adam (Kingma and Ba) is a popular optimizer in the training of deep neural networks. However, Reddi et al. have recently shown that the convergence proof of Adam is problematic and proposed a variant of Adam called AMSGrad as a fix. In this paper, we show that the convergence proof of AMSGrad is also problematic. Concretely, the problem in the convergence proof of AMSGrad is in handling the hyper-parameters, treating them as equal while they are not. This is also the neglected issue in the convergence proof of Adam. We provide an explicit counter-example of a simple convex optimization setting to show this neglected issue. Depending on manipulating the hyper-parameters, we present various fixes for this issue. We provide a new convergence proof for AMSGrad as the first fix. We also propose a new version of AMSGrad called AdamX as another fix. Our experiments on the benchmark dataset also support our theoretical results.