mlsanityData to 14 Oct 2026

ODE approximation for the Adam algorithm: General and overparametrized setting

arXiv 2511.04622 ↗First posted 6 Nov 2025math.OC cs.LG math.PRcited by 10

Not on the map. Why?

Abstract (arXiv)

The Adam optimizer is currently presumably the most popular optimization method in deep learning. In this article we develop an ODE based method to study the Adam optimizer in a fast-slow scaling regime. For fixed momentum parameters and vanishing step-sizes, we show that the Adam algorithm is an asymptotic pseudo-trajectory of the flow of a particular vector field, which is referred to as the Adam vector field. Leveraging properties of asymptotic pseudo-trajectories, we establish convergence results for the Adam algorithm. In particular, in a very general setting we show that if the Adam algorithm converges, then the limit must be a zero of the Adam vector field, rather than a local minimizer or critical point of the objective function. In contrast, in the overparametrized empirical risk minimization setting, the Adam algorithm is able to locally find the set of minima. Specifically, we show that in a neighborhood of the global minima, the objective function serves as a Lyapunov function for the flow induced by the Adam vector field. As a consequence, if the Adam algorithm enters a neighborhood of the global minima infinitely often, it converges to the set of global minima.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 28 of its 46 references that are in the map. First posted on arXiv 6 Nov 2025.

Builds on

28 of 46 references.

References · 28

  1. 2016An overview of gradient descent optimization algorithmscited by 1,033
  2. 2019On the Convergence of Adam and Beyondcited by 841
  3. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  4. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  5. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  6. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  7. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
  8. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  9. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  10. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  11. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 80
  12. 2013Fast Convergence of Stochastic Gradient Descent under a Strong Growth Conditioncited by 54
16 more
  1. 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 49
  2. 2023Convex and Non-convex Optimization Under Generalized Smoothnesscited by 44
  3. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 39
  4. 2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 33
  5. 2019Continuous Time Analysis of Momentum Methodscited by 32
  6. 2023Adam-family Methods for Nonsmooth Optimization with Convergence Guaranteescited by 22
  7. 2023Mathematical Introduction to Deep Learning: Methods, Implementations, and Theorycited by 20
  8. 2024Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning ratescited by 15
  9. 2024Convergence rates for the Adam optimizercited by 15
  10. 2023Exponential convergence rates for momentum stochastic gradient descent in the overparametrized settingcited by 8
  11. 2023Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descentcited by 7
  12. 2023Backward error analysis and the qualitative behaviour of stochastic optimization algorithms: Application to stochastic coordinate descentcited by 5
  13. 2023Adam-family Methods with Decoupled Weight Decay in Deep Learningcited by 5
  14. 2023Convergence of stochastic gradient descent under a local Lojasiewicz condition for deep neural networkscited by 3
  15. 2024Stochastic Modified Flows for Riemannian Stochastic Gradient Descentcited by 2
  16. 2025Effective continuous equations for adaptive SGD: a stochastic analysis viewcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Towards Understanding Adam Convergence on Highly Degenerate Polynomialscited by 4
  2. 2026Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization methodcited by 3
  3. 2026A Rod Flow Model for Adam at the Edge of Stabilitycited by 3
  4. 2026Deterministic Adam-Inspired Methods with Accelerated Convergence Ratecited by 1
  5. 2026Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimizationcited by 1
  6. 2026Central limit theorem for the averaged Adam optimizercited by 1
  7. 2026Unified convergence analysis for gradient descent optimization methods in the training of deep neural networkscited by 1
  8. 2026Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Expertscited by 0
  9. 2026The Role of Target Update Frequencies in Q-Learningcited by 0
  10. 2026Strong Averaging Principle and Long-Time Dynamics for Fast-Slow SDEs with Increasing Time-Scale Separation and Degenerate Noisecited by 0