# Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimization

arXiv: https://arxiv.org/abs/1810.02263 · HTML: https://mlsanity.com/paper/1810.02263/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2018.

## Built on by (most cited first, 30 of 49)

- [Adaptive Gradient Methods at the Edge of Stability](https://mlsanity.com/paper/2207.14484/) (2022)
- [The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon](https://mlsanity.com/paper/2206.04817/) (2022)
- [Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective](https://mlsanity.com/paper/2502.00604/) (2025)
- [SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain](https://mlsanity.com/paper/2407.19584/) (2024)
- [On the Implicit Bias of Adam](https://mlsanity.com/paper/2309.00079/) (2023)
- [Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees](https://mlsanity.com/paper/2305.03938/) (2023)
- [Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis](https://mlsanity.com/paper/2210.15898/) (2022)
- [Cautious Weight Decay](https://mlsanity.com/paper/2510.12402/) (2025)
- [The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks](https://mlsanity.com/paper/2602.16340/) (2026)
- [ODE approximation for the Adam algorithm: General and overparametrized setting](https://mlsanity.com/paper/2511.04622/) (2025)
- [Sharp higher order convergence rates for the Adam optimizer](https://mlsanity.com/paper/2504.19426/) (2025)
- [Regularized DeepIV with Model Selection](https://mlsanity.com/paper/2403.04236/) (2024)
- [Transformative or Conservative? Conservation laws for ResNets and Transformers](https://mlsanity.com/paper/2506.06194/) (2025)
- [Efficiency Ordering of Stochastic Gradient Descent](https://mlsanity.com/paper/2209.07446/) (2022)
- [How Memory in Optimization Algorithms Implicitly Modifies the Loss](https://mlsanity.com/paper/2502.02132/) (2025)
- [Adam-family Methods with Decoupled Weight Decay in Deep Learning](https://mlsanity.com/paper/2310.08858/) (2023)
- [Extensions of Robbins-Siegmund Theorem with Applications in Reinforcement Learning](https://mlsanity.com/paper/2509.26442/) (2025)
- [Continuous-Time Analysis of Adaptive Optimization and Normalization](https://mlsanity.com/paper/2411.05746/) (2024)
- [Towards Understanding Adam Convergence on Highly Degenerate Polynomials](https://mlsanity.com/paper/2603.09581/) (2026)
- [Beyond Mahalanobis-Based Scores for Textual OOD Detection](https://mlsanity.com/paper/2211.13527/) (2022)
- [Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimization](https://mlsanity.com/paper/2307.10053/) (2023)
- [Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method](https://mlsanity.com/paper/2603.18899/) (2026)
- [Generalized AdaGrad (G-AdaGrad) and Adam: A State-Space Perspective](https://mlsanity.com/paper/2106.00092/) (2021)
- [Accelerating Distributed Stochastic Optimization via Self-Repellent Random Walks](https://mlsanity.com/paper/2401.09665/) (2024)
- [A Deep Neural Network Framework for Solving Forward and Inverse Problems in Delay Differential Equations](https://mlsanity.com/paper/2408.09202/) (2024)
- [A Polyak-Ruppert Central Limit Theorem for SA-Adam with Momentum and Non-Convergent Adaptive Preconditioning](https://mlsanity.com/paper/2606.17364/) (2026)
- [A Control Theoretic Framework for Adaptive Gradient Optimizers in Machine Learning](https://mlsanity.com/paper/2206.02034/) (2022)
- [PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning](https://mlsanity.com/paper/2505.22085/) (2025)
- [From Adam to Adam-Like Lagrangians: Second-Order Nonlocal Dynamics](https://mlsanity.com/paper/2602.09101/) (2026)
- [Deterministic Adam-Inspired Methods with Accelerated Convergence Rate](https://mlsanity.com/paper/2604.08742/) (2026)

## Abstract (arXiv)

Adam is a popular variant of stochastic gradient descent for finding a local minimizer of a function. In the constant stepsize regime, assuming that the objective function is differentiable and non-convex, we establish the convergence in the long run of the iterates to a stationary point under a stability condition. The key ingredient is the introduction of a continuous-time version of Adam, under the form of a non-autonomous ordinary differential equation. This continuous-time system is a relevant approximation of the Adam iterates, in the sense that the interpolated Adam process converges weakly towards the solution to the ODE. The existence and the uniqueness of the solution are established. We further show the convergence of the solution towards the critical points of the objective function and quantify its convergence rate under a Lojasiewicz assumption. Then, we introduce a novel decreasing stepsize version of Adam. Under mild assumptions, it is shown that the iterates are almost surely bounded and converge almost surely to critical points of the objective function. Finally, we analyze the fluctuations of the algorithm by means of a conditional central limit theorem.