mlsanityData to 14 Oct 2026

A Theoretical and Empirical Study on the Convergence of Adam with an "Exact" Constant Step Size in Non-Convex Settings

arXiv 2309.08339 ↗First posted 15 Sep 2023cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

In neural network training, RMSProp and Adam remain widely favoured optimisation algorithms. One of the keys to their performance lies in selecting the correct step size, which can significantly influence their effectiveness. Additionally, questions about their theoretical convergence properties continue to be a subject of interest. In this paper, we theoretically analyse a constant step size version of Adam in the non-convex setting and discuss why it is important for the convergence of Adam to use a fixed step size. This work demonstrates the derivation and effective implementation of a constant step size for Adam, offering insights into its performance and efficiency in non convex optimisation scenarios. (i) First, we provide proof that these adaptive gradient algorithms are guaranteed to reach criticality for smooth non-convex objectives with constant step size, and we give bounds on the running time. Both deterministic and stochastic versions of Adam are analysed in this paper. We show sufficient conditions for the derived constant step size to achieve asymptotic convergence of the gradients to zero with minimal assumptions. Next, (ii) we design experiments to empirically study Adam's convergence with our proposed constant step size against stateof the art step size schedulers on classification tasks. Lastly, (iii) we also demonstrate that our derived constant step size has better abilities in reducing the gradient norms, and empirically, we show that despite the accumulation of a few past gradients, the key driver for convergence in Adam is the non-increasing step sizes.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 47 references that are in the map. First posted on arXiv 15 Sep 2023.

Builds on

27 of 47 references.

References · 27

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2021Convex optimizationcited by 2,515
  5. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  6. 2019On the Convergence of Adam and Beyondcited by 841
  7. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  8. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  9. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  10. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  11. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
  12. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
15 more
  1. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 293
  2. 2020Contrastive Clusteringcited by 265
  3. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  4. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  5. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  6. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  7. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
  8. 2020Lipschitz constant estimation of Neural Networks via sparse polynomial optimizationcited by 87
  9. 2015Beyond Convexity: Stochastic Quasi-Convex Optimizationcited by 76
  10. 2018Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov accelerationcited by 39
  11. 2020Implicit Rank-Minimizing Autoencodercited by 35
  12. 2021Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Accelerationcited by 25
  13. 2018AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methodscited by 24
  14. 2015Stop Wasting My Gradients: Practical SVRGcited by 14
  15. 2022Amos: An Adam-style Optimizer with Adaptive Weight Decay towards Model-Oriented Scalecited by 7

Built on by

Not cited yet.