mlsanityData to 14 Oct 2026

Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case

arXiv 2307.11782 ↗First posted 20 Jul 2023math.OC cs.LG cs.NAcited by 14

Not on the map. Why?

Abstract (arXiv)

Adam is a commonly used stochastic optimization algorithm in machine learning. However, its convergence is still not fully understood, especially in the non-convex setting. This paper focuses on exploring hyperparameter settings for the convergence of vanilla Adam and tackling the challenges of non-ergodic convergence related to practical application. The primary contributions are summarized as follows: firstly, we introduce precise definitions of ergodic and non-ergodic convergence, which cover nearly all forms of convergence for stochastic optimization algorithms. Meanwhile, we emphasize the superiority of non-ergodic convergence over ergodic convergence. Secondly, we establish a weaker sufficient condition for the ergodic convergence guarantee of Adam, allowing a more relaxed choice of hyperparameters. On this basis, we achieve the almost sure ergodic convergence rate of Adam, which is arbitrarily close to o(1/√K). More importantly, we prove, for the first time, that the last iterate of Adam converges to a stationary point for non-convex objectives. Finally, we obtain the non-ergodic convergence rate of O(1/K) for function values under the Polyak-Lojasiewicz (PL) condition. These findings build a solid theoretical foundation for Adam to solve non-convex stochastic optimization problems.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 50 of its 82 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

50 of 82 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  4. 2016An overview of gradient descent optimization algorithmscited by 1,033
  5. 2019On the Convergence of Adam and Beyondcited by 841
  6. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  7. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
  8. 2017How to Escape Saddle Points Efficientlycited by 285
  9. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  10. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  11. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  12. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
38 more
  1. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  2. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  3. 2020Better Theory for SGD in the Nonconvex Worldcited by 158
  4. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
  5. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  6. 2017Stability and Generalization of Learning Algorithms that Converge to Global Optimacited by 100
  7. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  8. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  9. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 90
  10. 2013Accelerated Gradient Methods for Nonconvex Nonlinear and Stochastic Programmingcited by 75
  11. 2020A Unified Convergence Analysis for Shuffling-Type Gradient Methodscited by 58
  12. 2023Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptionscited by 57
  13. 2017Variants of RMSProp and Adagrad with Logarithmic Regret Boundscited by 50
  14. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  15. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
  16. 2021SUPER-ADAM: Faster and Universal Framework of Adaptive Gradientscited by 40
  17. 2022High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsizecited by 40
  18. 2022On Almost Sure Convergence Rates of Stochastic Gradient Methodscited by 35
  19. 2018On exponential convergence of SGD in non-convex over-parametrized learningcited by 33
  20. 2018New Convergence Aspects of Stochastic Gradient Algorithmscited by 30
  21. 2022Provable Adaptivity of Adam under Non-uniform Smoothnesscited by 30
  22. 2017First-order Stochastic Algorithms for Escaping From Saddle Points in Almost Linear Timecited by 29
  23. 2019Linear Convergence of Adaptive Stochastic Gradient Descentcited by 29
  24. 2021Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimizationcited by 28
  25. 2022Revisiting Optimal Convergence Rate for Smooth and Non-convex Stochastic Decentralized Optimizationcited by 23
  26. 2020Asymptotic study of stochastic adaptive algorithm in non-convex landscapecited by 16
  27. 2018A Unified Analysis of AdaGrad with Weighted Aggregation and Momentum Accelerationcited by 15
  28. 2018Non-ergodic Convergence Analysis of Heavy-Ball Algorithmscited by 14
  29. 2022A Unified Convergence Theorem for Stochastic Optimization Methodscited by 12
  30. 2021Active manifolds, stratifications, and convergence to local minima in nonsmooth optimizationcited by 11
  31. 2021On the Convergence of Stochastic Gradient Descent with Bandwidth-based Step Sizecited by 9
  32. 2022Last-iterate convergence analysis of stochastic momentum methods for neural networkscited by 5
  33. 2022Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimizationcited by 5
  34. 2013Efficient Distance Metric Learning by Adaptive Sampling and Mini-Batch Stochastic Gradient Descent (SGD)cited by 4
  35. 2020Revisiting SGD with Increasingly Weighted Averaging: Optimization and Generalization Perspectivescited by 3
  36. 2020Variance Reduction on General Adaptive Stochastic Mirror Descentcited by 2
  37. 2022Stochastic Gradient Descent with Dependent Data for Offline Reinforcement Learningcited by 2
  38. 2022Scaling up Stochastic Gradient Descent for Non-convex Optimisationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 4.

14 papers.

Sort:

By area:

  1. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 38
  2. 2025On the Convergence of Muon and Beyondcited by 31
  3. 2024MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergencecited by 19
  4. 2025World Models That Know When They Don't Know - Controllable Video Generation with Calibrated Uncertaintycited by 17
  5. 2024Convergence rates for the Adam optimizercited by 15
  6. 2024On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyondcited by 14
  7. 2024A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGDcited by 9
  8. 2025Sharp higher order convergence rates for the Adam optimizercited by 8
  9. 2025AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-trainingcited by 6
  10. 2025Adaptive Regularization for Large-Scale Sparse Feature Embedding Modelscited by 2
  11. 2026MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimizationcited by 2
  12. 2026Unified convergence analysis for gradient descent optimization methods in the training of deep neural networkscited by 1
2 more
  1. 2026OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimalitycited by 0
  2. 2026Neural Network Implementation of the Renormalization Group for Fault Diagnosis with Class Imbalancecited by 0