mlsanityData to 14 Oct 2026

On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization

arXiv 1808.02941 ↗First posted 8 Aug 2018cs.LG math.OC stat.MLcited by 174

Not on the map. Why?

Abstract (arXiv)

This paper studies a class of adaptive gradient based momentum algorithms that update the search directions and learning rates simultaneously using past gradients. This class, which we refer to as the "Adam-type", includes the popular algorithms such as the Adam, AMSGrad and AdaGrad. Despite their popularity in training deep neural networks, the convergence of these algorithms for solving nonconvex problems remains an open question. This paper provides a set of mild sufficient conditions that guarantee the convergence for the Adam-type methods. We prove that under our derived conditions, these methods can achieve the convergence rate of order O(logT/√T) for nonconvex stochastic optimization. We show the conditions are essential in the sense that violating them may make the algorithm diverge. Moreover, we propose and analyze a class of (deterministic) incremental adaptive gradient algorithms, which has the same O(logT/√T) convergence rate. Our study could also be extended to a broader class of adaptive gradient methods in machine learning and optimization.

Key facts. Cited by 174 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 38; 2022: 33; 2023: 23; 2024: 31; 2025: 25; 2026: 24.

174 papers.

Sort:

By area:

  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  2. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
  3. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  4. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  5. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  6. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  7. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 81
  8. 2022The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variancecited by 61
  9. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  10. 2023A Theory on Adam Instability in Large-Scale Machine Learningcited by 56
  11. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  12. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 53
38 more
  1. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  2. 2022Understanding AdamW through Proximal Methods and Scale-Freenesscited by 46
  3. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 46
  4. 2021SUPER-ADAM: Faster and Universal Framework of Adaptive Gradientscited by 40
  5. 2022High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsizecited by 40
  6. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  7. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 38
  8. 2024The Implicit Bias of Adam on Separable Datacited by 36
  9. 2021BiAdam: Fast Adaptive Bilevel Optimization Methodscited by 33
  10. 2025On the Convergence of Muon and Beyondcited by 31
  11. 2022Provable Adaptivity of Adam under Non-uniform Smoothnesscited by 30
  12. 2024Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguisecited by 26
  13. 2021Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Accelerationcited by 25
  14. 2021Toward Communication Efficient Adaptive Gradient Methodcited by 25
  15. 2023Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methodscited by 23
  16. 2024Adam with model exponential moving average is effective for nonconvex optimizationcited by 23
  17. 2024Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivitycited by 23
  18. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 22
  19. 2023AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networkscited by 21
  20. 2024On Convergence of Adam for Stochastic Optimization under Relaxed Assumptionscited by 21
  21. 2022On Distributed Adaptive Optimization with Gradient Compressioncited by 19
  22. 2024Towards Quantifying the Preconditioning Effect of Adamcited by 19
  23. 2024ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Ratecited by 18
  24. 2021On the Convergence of Decentralized Adaptive Gradient Methodscited by 17
  25. 2025World Models That Know When They Don't Know - Controllable Video Generation with Calibrated Uncertaintycited by 17
  26. 2022Nest Your Adaptive Algorithm for Parameter-Agnostic Nonconvex Minimax Optimizationcited by 16
  27. 2024Provable Complexity Improvement of AdaGrad over SGD: Upper and Lower Bounds in Stochastic Non-Convex Optimizationcited by 16
  28. 2022Maximizing Communication Efficiency for Large-scale Training via 0/1 Adamcited by 15
  29. 2023FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clusteringcited by 15
  30. 2024Convergence rates for the Adam optimizercited by 15
  31. 2023Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Casecited by 14
  32. 2024Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variancecited by 14
  33. 2024AdaFisher: Adaptive Second Order Optimization via Fisher Informationcited by 14
  34. 2021Unified Convergence Analysis for Adaptive Optimization with Moving Average Estimatorcited by 13
  35. 2021AdaGDA: Faster Adaptive Gradient Descent Ascent Methods for Minimax Optimizationcited by 13
  36. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 13
  37. 2025Cautious Weight Decaycited by 13
  38. 2021Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimizationcited by 12

50 most cited of 174 All 174 (JSON)