mlsanityData to 7 Oct 2026

On the Convergence of Adam and Beyond

arXiv 1904.09237 ↗First posted 19 Apr 2019cs.LG math.OC stat.MLcited by 834

Not on the map. Why?

Abstract (arXiv)

Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large output spaces, it has been empirically observed that these algorithms fail to converge to an optimal solution (or a critical point in nonconvex settings). We show that one cause for such failures is the exponential moving average used in the algorithms. We provide an explicit example of a simple convex optimization setting where Adam does not converge to the optimal solution, and describe the precise problems with the previous analysis of Adam algorithm. Our analysis suggests that the convergence issues can be fixed by endowing such algorithms with `long-term memory' of past gradients, and propose new variants of the Adam algorithm which not only fix the convergence issues but often also lead to improved empirical performance.

Key facts. Cited by 834 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 171; 2022: 134; 2023: 133; 2024: 144; 2025: 121; 2026: 129.

834 papers.

Sort:

By area:

  1. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  2. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
  3. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  4. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 342
  5. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  6. 2022Learning Local Equivariant Representations for Large-Scale Atomistic Dynamicscited by 254
  7. 2021Robust Motion In-betweeningcited by 223
  8. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  9. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  10. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  11. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 159
  12. 2024The Road Less Scheduledcited by 158
38 more
  1. 2021Hyperbolic Deep Neural Networks: A Surveycited by 138
  2. 2022CFA: Coupled-hypersphere-based Feature Adaptation for Target-Oriented Anomaly Localizationcited by 135
  3. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  4. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  5. 20223D Common Corruptions and Data Augmentationcited by 120
  6. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  7. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  8. 2021Dive into Deep Learningcited by 111
  9. 2021SpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effectscited by 103
  10. 2022CARD: Classification and Regression Diffusion Modelscited by 102
  11. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
  12. 2023Convergence of Adam Under Relaxed Assumptionscited by 97
  13. 2023Hidden Biases of End-to-End Driving Modelscited by 95
  14. 2024Dinomaly: The Less Is More Philosophy in Multi-Class Unsupervised Anomaly Detectioncited by 92
  15. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  16. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
  17. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  18. 2023PivotNet: Vectorized Pivot Learning for End-to-end HD Map Constructioncited by 84
  19. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  20. 2022BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Predictioncited by 82
  21. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 81
  22. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  23. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
  24. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  25. 2021On Statistical Bias In Active Learning: How and When To Fix Itcited by 72
  26. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  27. 2021SE(3)-equivariant prediction of molecular wavefunctions and electronic densitiescited by 67
  28. 2022ViTPose++: Vision Transformer for Generic Body Pose Estimationcited by 65
  29. 2023Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimizationcited by 65
  30. 2024Scalable Optimization in the Modular Normcited by 64
  31. 2023ReContrast: Domain-Specific Anomaly Detection via Contrastive Reconstructioncited by 62
  32. 2022GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasetscited by 61
  33. 2024GuardT2I: Defending Text-to-Image Models from Adversarial Promptscited by 61
  34. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  35. 2025AdaMuon: Adaptive Muon Optimizercited by 59
  36. 2023EgoHumans: An Egocentric 3D Multi-Human Benchmarkcited by 58
  37. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  38. 2022Communication-Efficient Adaptive Federated Learningcited by 55

50 most cited of 834 All 834 (JSON)