mlsanityData to 14 Oct 2026

Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method

arXiv 2603.18899 ↗First posted 19 Mar 2026cs.LG math.OCcited by 3

Not on the map. Why?

Abstract (arXiv)

The adaptive moment estimation (Adam) optimizer proposed by Kingma & Ba (2014) is presumably the most popular stochastic gradient descent (SGD) optimization method for the training of deep neural networks (DNNs) in artificial intelligence (AI) systems. Despite its groundbreaking success in the training of AI systems, it still remains an open research problem to provide a complete error analysis of Adam, not only for optimizing DNNs but even when applied to strongly convex stochastic optimization problems (SOPs). Previous error analysis results for strongly convex SOPs in the literature provide conditional convergence analyses that rely on the assumption that Adam does not diverge to infinity but remains uniformly bounded. It is the key contribution of this work to establish uniform a priori bounds for Adam and, thereby, to provide -- for the first time -- an unconditional error analysis for Adam for a large class of strongly convex SOPs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

29 of 38 references.

References · 29

  1. 2016An overview of gradient descent optimization algorithmscited by 1,033
  2. 2019On the Convergence of Adam and Beyondcited by 841
  3. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  4. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
  5. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  6. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  7. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  8. 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 49
  9. 2018Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov accelerationcited by 39
  10. 2022Provable Adaptivity of Adam under Non-uniform Smoothnesscited by 30
  11. 2023Mathematical Introduction to Deep Learning: Methods, Implementations, and Theorycited by 20
  12. 2018An improvement of the convergence proof of the ADAM-Optimizercited by 19
17 more
  1. 2018A Unified Analysis of AdaGrad with Weighted Aggregation and Momentum Accelerationcited by 15
  2. 2024Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning ratescited by 15
  3. 2024Convergence rates for the Adam optimizercited by 15
  4. 2020Non-convergence of stochastic gradient descent in the training of deep neural networkscited by 14
  5. 2024On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyondcited by 14
  6. 2024Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variancecited by 14
  7. 2021Unified Convergence Analysis for Adaptive Optimization with Moving Average Estimatorcited by 13
  8. 2025ODE approximation for the Adam algorithm: General and overparametrized settingcited by 10
  9. 2022Non-Convergence and Limit Cycles in the Adam optimizercited by 9
  10. 2022Blow up phenomena for gradient descent optimization methods in the training of artificial neural networkscited by 9
  11. 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
  12. 2024Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activationcited by 8
  13. 2025Sharp higher order convergence rates for the Adam optimizercited by 8
  14. 2025Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networkscited by 6
  15. 2025Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizercited by 6
  16. 2025Asymptotic stability properties and a priori bounds for Adam and other gradient descent optimization methodscited by 4
  17. 2019Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Central limit theorem for the averaged Adam optimizercited by 1
  2. 2026Unified convergence analysis for gradient descent optimization methods in the training of deep neural networkscited by 1
  3. 2026Convergence rates for the RMSprop optimizer with full control of the hyperparameterscited by 0