mlsanityData to 14 Oct 2026

Convergence of Adam Under Relaxed Assumptions

arXiv 2304.13972 ↗First posted 27 Apr 2023math.OC cs.LG stat.MLcited by 99

Not on the map. Why?

Abstract (arXiv)

In this paper, we provide a rigorous proof of convergence of the Adaptive Moment Estimate (Adam) algorithm for a wide class of optimization objectives. Despite the popularity and efficiency of the Adam algorithm in training deep neural networks, its theoretical properties are not yet fully understood, and existing convergence proofs require unrealistically strong assumptions, such as globally bounded gradients, to show the convergence to stationary points. In this paper, we show that Adam provably converges to ε-stationary points with O(ε-4) gradient complexity under far more realistic conditions. The key to our analysis is a new proof of boundedness of gradients along the optimization trajectory of Adam, under a generalized smoothness assumption according to which the local smoothness (i.e., Hessian norm when it exists) is bounded by a sub-quadratic function of the gradient norm. Moreover, we propose a variance-reduced version of Adam with an accelerated gradient complexity of O(ε-3).

Key facts. Cited by 99 papers in the map (data through 14 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 27 Apr 2023.

Builds on

42 of 56 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2017Proximal Policy Optimization Algorithmscited by 15,215
  7. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  8. 2015Continuous control with deep reinforcement learningcited by 3,407
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  10. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  11. 2019On the Convergence of Adam and Beyondcited by 841
  12. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
30 more
  1. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  2. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
  3. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  4. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 287
  5. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  6. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  7. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  8. 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 145
  9. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
  10. 2020PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimizationcited by 119
  11. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  12. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  13. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 89
  14. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 81
  15. 2016Variance Reduction for Faster Non-Convex Optimizationcited by 72
  16. 2017Non-convex Finite-Sum Optimization Via SCSG Methodscited by 61
  17. 2022The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variancecited by 61
  18. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  19. 2023Beyond Uniform Smoothness: A Stopped Analysis of Adaptive SGDcited by 41
  20. 2018Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov accelerationcited by 39
  21. 2023Generalized-Smooth Nonconvex Optimization is As Efficient As Smooth Nonconvex Optimizationcited by 35
  22. 2019Hybrid Stochastic Gradient Descent Algorithms for Stochastic Nonconvex Optimizationcited by 32
  23. 2022Provable Adaptivity of Adam under Non-uniform Smoothnesscited by 30
  24. 2021Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Accelerationcited by 25
  25. 2018AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methodscited by 24
  26. 2023Variance-reduced Clipping for Non-convex Optimizationcited by 22
  27. 2020Asymptotic study of stochastic adaptive algorithm in non-convex landscapecited by 16
  28. 2020An Optimal Hybrid Variance-Reduced Algorithm for Stochastic Composite Nonconvex Optimizationcited by 10
  29. 2022Divergence Results and Convergence of a Variance Reduced Version of ADAMcited by 8
  30. 2022Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smoothnesscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 35; 2025: 27; 2026: 31.

99 papers.

Sort:

By area:

  1. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  2. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
  3. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 64
  4. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 46
  5. 2023Convex and Non-convex Optimization Under Generalized Smoothnesscited by 44
  6. 2024Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clippingcited by 42
  7. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  8. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 38
  9. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  10. 2025On the Convergence of Muon and Beyondcited by 31
  11. 2023Parameter-Agnostic Optimization under Relaxed Smoothnesscited by 29
  12. 2024BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Modelscited by 27
38 more
  1. 2024Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguisecited by 26
  2. 2024Adam with model exponential moving average is effective for nonconvex optimizationcited by 23
  3. 2024Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivitycited by 23
  4. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  5. 2024On Convergence of Adam for Stochastic Optimization under Relaxed Assumptionscited by 21
  6. 2024ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Ratecited by 18
  7. 2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 17
  8. 2024Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysiscited by 15
  9. 2024Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning ratescited by 15
  10. 2024Convergence rates for the Adam optimizercited by 15
  11. 2023Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Casecited by 14
  12. 2024On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyondcited by 14
  13. 2024Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variancecited by 14
  14. 2025Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under (L0, L1)-Smoothnesscited by 13
  15. 2025Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelismcited by 12
  16. 2024Understanding Adam Requires Better Rotation Dependent Assumptionscited by 11
  17. 2024Convergence Rate Analysis of LIONcited by 11
  18. 2025ODE approximation for the Adam algorithm: General and overparametrized settingcited by 10
  19. 2024An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothnesscited by 9
  20. 2024A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGDcited by 9
  21. 2023High Probability Convergence of Adam Under Unbounded Gradients and Affine Variance Noisecited by 8
  22. 2024No More Adam: Learning Rate Scaling at Initialization is All You Needcited by 8
  23. 2025Sharp higher order convergence rates for the Adam optimizercited by 8
  24. 2025On the O(frac√dK1/4) Convergence Rate of AdamW Measured by ℓ1 Normcited by 8
  25. 2024MGDA Converges under Generalized Smoothness, Provablycited by 7
  26. 2024μLO: Compute-Efficient Meta-Generalization of Learned Optimizerscited by 7
  27. 2024A Nearly Optimal Single Loop Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothnesscited by 7
  28. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 6
  29. 2025Low-rank bias, weight decay, and model merging in neural networkscited by 5
  30. 2025Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamicscited by 5
  31. 2024Adaptive Gradient Normalization and Independent Sampling for (Stochastic) Generalized-Smooth Optimizationcited by 4
  32. 2024Error Feedback under (L0,L1)-Smoothness: Normalization and Momentumcited by 4
  33. 2024Grams: Gradient Descent with Adaptive Momentum Scalingcited by 4
  34. 2025On the Interaction of Batch Noise, Adaptivity, and Compression, under (L0,L1)-Smoothness: An SDE Approachcited by 4
  35. 2025Simple Convergence Proof of Adam From a Sign-like Descent Perspectivecited by 4
  36. 2025Theoretical Analysis on how Learning Rate Warmup Accelerates Convergencecited by 4
  37. 2026Dynamic Regret via Discounted-to-Dynamic Reduction with Applications to Curved Losses and Adam Optimizercited by 4
  38. 2025SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximationcited by 3

50 most cited of 99 All 99 (JSON)