mlsanityData to 7 Oct 2026

A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD

arXiv 2410.04458 ↗First posted 6 Oct 2024cs.LG math.OCcited by 9

Not on the map. Why?

Abstract (arXiv)

Adaptive Moment Estimation (Adam) is a cornerstone optimization algorithm in deep learning, widely recognized for its flexibility with adaptive learning rates and efficiency in handling large-scale data. However, despite its practical success, the theoretical understanding of Adam's convergence has been constrained by stringent assumptions, such as almost surely bounded stochastic gradients or uniformly bounded gradients, which are more restrictive than those typically required for analyzing stochastic gradient descent (SGD). In this paper, we introduce a novel and comprehensive framework for analyzing the convergence properties of Adam. This framework offers a versatile approach to establishing Adam's convergence. Specifically, we prove that Adam achieves asymptotic (last iterate sense) convergence in both the almost sure sense and the (L1) sense under the relaxed assumptions typically used for SGD, namely (L)-smoothness and the ABC inequality. Meanwhile, under the same assumptions, we show that Adam attains non-asymptotic sample complexity bounds similar to those of SGD.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 13 of its 18 references that are in the map. First posted on arXiv 6 Oct 2024.

Builds on

13 of 18 references.

References · 13

  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  2. 2019On the Convergence of Adam and Beyondcited by 834
  3. 2020Better Theory for SGD in the Nonconvex Worldcited by 156
  4. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
  5. 2020An Improved Analysis of Stochastic Gradient Descent with Momentumcited by 117
  6. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  7. 2023Convergence of Adam Under Relaxed Assumptionscited by 97
  8. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 94
  9. 2023Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptionscited by 55
  10. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  11. 2022Provable Adaptivity of Adam under Non-uniform Smoothnesscited by 30
  12. 2023Adam-family Methods for Nonsmooth Optimization with Convergence Guaranteescited by 22
1 more
  1. 2023Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Casecited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedbackcited by 11
  2. 2025On the O(frac√dK1/4) Convergence Rate of AdamW Measured by ℓ1 Normcited by 8
  3. 2025How to Set β1, β2 in Adam: An Online Learning Perspectivecited by 2
  4. 2025Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regimecited by 2
  5. 2026Second-Moment Memory in Coordinatewise Adamcited by 1
  6. 2026HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalizationcited by 0
  7. 2026Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimizationcited by 0
  8. 2026Analysis of Adam Algorithms for Stochastic Dynamic Systemscited by 0
  9. 2026Adam under Generalized Smoothness with Second-Moment-Type Stochastic Gradientscited by 0