mlsanityData to 7 Oct 2026

On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization

arXiv 1808.05671 ↗First posted 16 Aug 2018cs.LG math.OC stat.MLcited by 91

Not on the map. Why?

Abstract (arXiv)

Adaptive gradient methods are workhorses in deep learning. However, the convergence guarantees of adaptive gradient methods for nonconvex optimization have not been thoroughly studied. In this paper, we provide a fine-grained convergence analysis for a general class of adaptive gradient methods including AMSGrad, RMSProp and AdaGrad. For smooth nonconvex functions, we prove that adaptive gradient methods in expectation converge to a first-order stationary point. Our convergence rate is better than existing results for adaptive gradient methods in terms of dimension. In addition, we also prove high probability bounds on the convergence rates of AMSGrad, RMSProp as well as AdaGrad, which have not been established before. Our analyses shed light on better understanding the mechanism behind adaptive gradient methods in optimizing nonconvex objectives.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 23; 2023: 18; 2024: 21; 2025: 7; 2026: 11.

91 papers.

Sort:

  1. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  2. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  3. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  4. 2023Convergence of Adam Under Relaxed Assumptionscited by 97
  5. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  6. 2022The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variancecited by 60
  7. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  8. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 50
  9. 2022Understanding AdamW through Proximal Methods and Scale-Freenesscited by 46
  10. 2023Towards Understanding the Generalization of Graph Neural Networkscited by 41
  11. 2022High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsizecited by 40
  12. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 38
38 more
  1. 2022Adapting to Mixing Time in Stochastic Optimization with Markovian Datacited by 35
  2. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 35
  3. 2024The Implicit Bias of Adam on Separable Datacited by 32
  4. 2022Provable Adaptivity of Adam under Non-uniform Smoothnesscited by 30
  5. 2023SGD with AdaGrad Stepsizes: Full Adaptivity with High Probability to Unknown Parameters, Unbounded Gradients and Affine Variancecited by 30
  6. 2025On the Convergence of Muon and Beyondcited by 28
  7. 2021Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimizationcited by 27
  8. 2021Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Accelerationcited by 25
  9. 2022Stability and Generalization of Stochastic Optimization with Nonconvex and Nonsmooth Problemscited by 25
  10. 2023Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methodscited by 23
  11. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  12. 2023AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networkscited by 21
  13. 2024On Convergence of Adam for Stochastic Optimization under Relaxed Assumptionscited by 21
  14. 2024Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivitycited by 20
  15. 2022On Distributed Adaptive Optimization with Gradient Compressioncited by 19
  16. 2024Towards Quantifying the Preconditioning Effect of Adamcited by 19
  17. 2024MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergencecited by 19
  18. 2024ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Ratecited by 18
  19. 2021On the Convergence of Decentralized Adaptive Gradient Methodscited by 17
  20. 2022Maximizing Communication Efficiency for Large-scale Training via 0/1 Adamcited by 15
  21. 2024Revisiting Convergence of AdaGrad with Relaxed Assumptionscited by 14
  22. 2023Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Casecited by 13
  23. 2021Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimizationcited by 12
  24. 2024Understanding Adam Requires Better Rotation Dependent Assumptionscited by 10
  25. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 10
  26. 2021On the One-sided Convergence of Adam-type Algorithms in Non-convex Non-concave Min-max Optimizationcited by 9
  27. 2022A Projection-free Algorithm for Constrained Stochastic Multi-level Composition Optimizationcited by 8
  28. 2023High Probability Convergence of Adam Under Unbounded Gradients and Affine Variance Noisecited by 8
  29. 2024AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methodscited by 8
  30. 2022TiAda: A Time-scale Adaptive Algorithm for Nonconvex Minimax Optimizationcited by 7
  31. 2021Frequency-aware SGD for Efficient Embedding Learning with Provable Benefitscited by 6
  32. 2023Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Trainingcited by 6
  33. 2024Regularized DeepIV with Model Selectioncited by 6
  34. 2022Communication-Efficient TeraByte-Scale Model Training Framework for Online Advertisingcited by 5
  35. 2022On the Algorithmic Stability and Generalization of Adaptive Optimization Methodscited by 5
  36. 2023High Probability Analysis for Non-Convex Stochastic Optimization with Clippingcited by 5
  37. 2021Accelerating Federated Learning with a Global Biased Optimisercited by 4
  38. 2022Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smoothnesscited by 4

50 most cited of 91 All 91 (JSON)