mlsanityData to 7 Oct 2026

Local Convergence of Adaptive Gradient Descent Optimizers

arXiv 2102.09804 ↗First posted 19 Feb 2021cs.LG cs.NA math.NAcited by 4

Not on the map. Why?

Abstract (arXiv)

Adaptive Moment Estimation (ADAM) is a very popular training algorithm for deep neural networks and belongs to the family of adaptive gradient descent optimizers. However to the best of the authors knowledge no complete convergence analysis exists for ADAM. The contribution of this paper is a method for the local convergence analysis in batch mode for a deterministic fixed training set, which gives necessary conditions for the hyperparameters of the ADAM algorithm. Due to the local nature of the arguments the objective function can be non-convex but must be at least twice continuously differentiable. Then we apply this procedure to other adaptive gradient descent algorithms and show for most of them local convergence with hyperparameter bounds.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 12 of its 29 references that are in the map. First posted on arXiv 19 Feb 2021.

Builds on

12 of 29 references.

References · 12

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2019On the Convergence of Adam and Beyondcited by 834
  5. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
  6. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 181
  7. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  8. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
  9. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 121
  10. 2018A general system of differential equations to model first order adaptive algorithmscited by 28
  11. 2018An improvement of the convergence proof of the ADAM-Optimizercited by 19
  12. 2018Lyapunov Theory for Discrete Time Systemscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 0; 2023: 0; 2024: 0; 2025: 0; 2026: 2.

4 papers.

  1. 2021Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Netscited by 4
  2. 2026Deterministic Adam-Inspired Methods with Accelerated Convergence Ratecited by 1
  3. 2021On a Conjecture Regarding the Adam Optimizercited by 0
  4. 2026Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approachcited by 0