mlsanityData to 7 Oct 2026

ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Rate

arXiv 2411.02853 ↗First posted 5 Nov 2024cs.LG stat.MLcited by 18

Not on the map. Why?

Abstract (arXiv)

Adam is one of the most popular optimization algorithms in deep learning. However, it is known that Adam does not converge in theory unless choosing a hyperparameter, i.e., β2, in a problem-dependent manner. There have been many attempts to fix the non-convergence (e.g., AMSGrad), but they require an impractical assumption that the gradient noise is uniformly bounded. In this paper, we propose a new adaptive gradient method named ADOPT, which achieves the optimal convergence rate of O ( 1 / √T ) with any choice of β2 without depending on the bounded noise assumption. ADOPT addresses the non-convergence issue of Adam by removing the current gradient from the second moment estimate and changing the order of the momentum update and the normalization by the second moment estimate. We also conduct intensive numerical experiments, and verify that our ADOPT achieves superior results compared to Adam and its variants across a wide range of tasks, including image classification, generative modeling, natural language processing, and deep reinforcement learning. The implementation is available at https://github.com/iShohei220/adopt.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 25 of its 41 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

25 of 41 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2019On the Convergence of Adam and Beyondcited by 834
  10. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  11. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  12. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
13 more
  1. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  2. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  3. 2020Better Theory for SGD in the Nonconvex Worldcited by 156
  4. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  5. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
  6. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  7. 2023Convergence of Adam Under Relaxed Assumptionscited by 97
  8. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  9. 2018Random Shuffling Beats SGD after Finite Epochscited by 66
  10. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  11. 2019The Complexity of Finding Stationary Points with Stochastic Gradient Descentcited by 34
  12. 2018AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methodscited by 24
  13. 2021On the Convergence of Step Decay Step-Size for Stochastic Optimizationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 10.

18 papers.

Sort:

  1. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 42
  2. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 10
  3. 2025On the O(frac√dK1/4) Convergence Rate of AdamW Measured by ℓ1 Normcited by 8
  4. 2024Grams: Gradient Descent with Adaptive Momentum Scalingcited by 3
  5. 2025DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Modelscited by 3
  6. 2026Benchmarking Optimizers for MLPs in Tabular Deep Learningcited by 3
  7. 2025How to Set β1, β2 in Adam: An Online Learning Perspectivecited by 2
  8. 2025MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updatescited by 2
  9. 2024Towards Simple and Provable Parameter-Free Adaptive Gradient Methodscited by 1
  10. 2026Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimizationcited by 1
  11. 2026Second-Moment Memory in Coordinatewise Adamcited by 1
  12. 2026Extending μP: Spectral Conditions for Feature Learning Across Optimizerscited by 0
6 more
  1. 2026HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalizationcited by 0
  2. 2026Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimizationcited by 0
  3. 2026AMO: Operator-level Adaptive Muon Orthogonalizationcited by 0
  4. 2026SUNTA: Hierarchical Video Prediction with Surprise-based Chunkingcited by 0
  5. 2026SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspacescited by 0
  6. 2026Blog: Survey of Optimizerscited by 0