mlsanityData to 14 Oct 2026

On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions

arXiv 2402.03982 ↗First posted 6 Feb 2024math.OC cs.LG stat.MLcited by 21

Not on the map. Why?

Abstract (arXiv)

The Adaptive Momentum Estimation (Adam) algorithm is highly effective in training various deep learning tasks. Despite this, there's limited theoretical understanding for Adam, especially when focusing on its vanilla form in non-convex smooth scenarios with potential unbounded gradients and affine variance noise. In this paper, we study vanilla Adam under these challenging conditions. We introduce a comprehensive noise model which governs affine variance noise, bounded noise and sub-Gaussian noise. We show that Adam can find a stationary point with a O(poly(log T)/√T) rate in high probability under this general noise model where T denotes total number iterations, matching the lower rate of stochastic first-order algorithms up to logarithm factors. More importantly, we reveal that Adam is free of tuning step-sizes with any problem-parameters, yielding a better adaptation property than the Stochastic Gradient Descent under the same conditions. We also provide a probabilistic convergence result for Adam under a generalized smooth condition which allows unbounded smoothness parameters and has been illustrated empirically to more accurately capture the smooth property of many practical objective functions.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 41 of its 65 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

41 of 65 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2016Densely Connected Convolutional Networkscited by 6,720
  5. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  6. 2019On the Convergence of Adam and Beyondcited by 841
  7. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  8. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  9. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
  10. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  11. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  12. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
29 more
  1. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
  2. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
  3. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  4. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  5. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  6. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  7. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 90
  8. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 89
  9. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 81
  10. 2014Global convergence of the Heavy-ball method for convex optimizationcited by 61
  11. 2020A High Probability Analysis of Adaptive SGD with Momentumcited by 61
  12. 2022The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variancecited by 61
  13. 2023Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptionscited by 57
  14. 2018A Unified Analysis of Stochastic Momentum Methods for Deep Learningcited by 55
  15. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  16. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  17. 2023High Probability Convergence of Stochastic Gradient Methodscited by 49
  18. 2023Beyond Uniform Smoothness: A Stopped Analysis of Adaptive SGDcited by 41
  19. 2021SUPER-ADAM: Faster and Universal Framework of Adaptive Gradientscited by 40
  20. 2022High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsizecited by 40
  21. 2018Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov accelerationcited by 39
  22. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  23. 2023SGD with AdaGrad Stepsizes: Full Adaptivity with High Probability to Unknown Parameters, Unbounded Gradients and Affine Variancecited by 30
  24. 2019Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Netscited by 29
  25. 2023Variance-reduced Clipping for Non-convex Optimizationcited by 22
  26. 2022Nest Your Adaptive Algorithm for Parameter-Agnostic Nonconvex Minimax Optimizationcited by 16
  27. 2018A Unified Analysis of AdaGrad with Weighted Aggregation and Momentum Accelerationcited by 15
  28. 2022Divergence Results and Convergence of a Variance Reduced Version of ADAMcited by 8
  29. 2023High Probability Convergence of Adam Under Unbounded Gradients and Affine Variance Noisecited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 11.

21 papers.

Sort:

  1. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 38
  2. 2024The Implicit Bias of Adam on Separable Datacited by 36
  3. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  4. 2024Revisiting Convergence of AdaGrad with Relaxed Assumptionscited by 15
  5. 2025On the O(frac√dK1/4) Convergence Rate of AdamW Measured by ℓ1 Normcited by 8
  6. 2024Continuous-Time Analysis of Adaptive Optimization and Normalizationcited by 4
  7. 2025Simple Convergence Proof of Adam From a Sign-like Descent Perspectivecited by 4
  8. 2026Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGradcited by 3
  9. 2025Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regimecited by 2
  10. 2026MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimizationcited by 2
  11. 2025Convergence of Adam in Deep ReLU Networks via Directional Complexity and Kakeya Boundscited by 1
  12. 2025SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adamcited by 1
9 more
  1. 2026Convergence Rate Analysis of the AdamW-style Shampoo: Unifying One-Sided and Two-Sided Preconditioningcited by 1
  2. 2026Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimizationcited by 1
  3. 2026Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?cited by 1
  4. 2026Analysis of Adam Algorithms for Stochastic Dynamic Systemscited by 1
  5. 2026Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approachcited by 0
  6. 2026Convergence rates for the RMSprop optimizer with full control of the hyperparameterscited by 0
  7. 2026Why Clipping Matters in AdaGrad? Toward a High-Probability Theory under Generalized Smoothnesscited by 0
  8. 2026Revisiting AdaGrad in Stochastic Convex Optimization: Last Iterates, High Probability, and Lower Boundscited by 0
  9. 2026Adam under Generalized Smoothness with Second-Moment-Type Stochastic Gradientscited by 0