mlsanityData to 14 Oct 2026

UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization

arXiv 2305.05675 ↗First posted 9 May 2023cs.LG cs.NA math.NAcited by 2

Not on the map. Why?

Abstract (arXiv)

Adam-type algorithms have become a preferred choice for optimisation in the deep learning setting, however, despite success, their convergence is still not well understood. To this end, we introduce a unified framework for Adam-type algorithms (called UAdam). This is equipped with a general form of the second-order moment, which makes it possible to include Adam and its variants as special cases, such as NAdam, AMSGrad, AdaBound, AdaFom, and Adan. This is supported by a rigorous convergence analysis of UAdam in the non-convex stochastic setting, showing that UAdam converges to the neighborhood of stationary points with the rate of O(1/T). Furthermore, the size of neighborhood decreases as β increases. Importantly, our analysis only requires the first-order momentum factor to be close enough to 1, without any restrictions on the second-order momentum factor. Theoretical results also show that vanilla Adam can converge by selecting appropriate hyperparameters, which provides a theoretical guarantee for the analysis, applications, and further developments of the whole class of Adam-type algorithms.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 23 of its 52 references that are in the map. First posted on arXiv 9 May 2023.

Builds on

23 of 52 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  5. 2019On the Convergence of Adam and Beyondcited by 841
  6. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  7. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  8. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  9. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  10. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
  11. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  12. 2020Better Theory for SGD in the Nonconvex Worldcited by 158
11 more
  1. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
  2. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  3. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
  4. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  5. 2014Stochastic Compositional Gradient Descent: Algorithms for Minimizing Compositions of Expected-Value Functionscited by 95
  6. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  7. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  8. 2022Provable Adaptivity of Adam under Non-uniform Smoothnesscited by 30
  9. 2021Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Accelerationcited by 25
  10. 2020Adaptivity of Stochastic Gradient Methods for Nonconvex Optimizationcited by 16
  11. 2022Last-iterate convergence analysis of stochastic momentum methods for neural networkscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimizationcited by 1
  2. 2026Unified convergence analysis for gradient descent optimization methods in the training of deep neural networkscited by 1