# ODE approximation for the Adam algorithm: General and overparametrized setting arXiv: https://arxiv.org/abs/2511.04622 ยท HTML: https://mlsanity.com/paper/2511.04622/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 46 references that are in the map. - First posted on arXiv 6 Nov 2025. ## Builds on (most cited first, 28 of 28) - [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016) - [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019) - [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019) - [The Marginal Value of Adaptive Gradient Methods in Machine Learning](https://mlsanity.com/paper/1705.08292/) (2017) - [Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptron](https://mlsanity.com/paper/1810.07288/) (2018) - [Stochastic modified equations and adaptive stochastic gradient algorithms](https://mlsanity.com/paper/1511.06251/) (2015) - [Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations](https://mlsanity.com/paper/1811.01558/) (2018) - [Adam Can Converge Without Any Modification On Update Rules](https://mlsanity.com/paper/2208.09632/) (2022) - [On the SDEs and Scaling Rules for Adaptive Gradient Algorithms](https://mlsanity.com/paper/2205.10287/) (2022) - [Convergence of Adam Under Relaxed Assumptions](https://mlsanity.com/paper/2304.13972/) (2023) - [On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problems](https://mlsanity.com/paper/2006.11144/) (2020) - [Fast Convergence of Stochastic Gradient Descent under a Strong Growth Condition](https://mlsanity.com/paper/1308.6370/) (2013) - [Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimization](https://mlsanity.com/paper/1810.02263/) (2018) - [Convex and Non-convex Optimization Under Generalized Smoothness](https://mlsanity.com/paper/2306.01264/) (2023) - [Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysis](https://mlsanity.com/paper/2105.01650/) (2021) - [A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms](https://mlsanity.com/paper/2009.06125/) (2020) - [Continuous Time Analysis of Momentum Methods](https://mlsanity.com/paper/1906.04285/) (2019) - [Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees](https://mlsanity.com/paper/2305.03938/) (2023) - [Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory](https://mlsanity.com/paper/2310.20360/) (2023) - [Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates](https://mlsanity.com/paper/2407.08100/) (2024) - [Convergence rates for the Adam optimizer](https://mlsanity.com/paper/2407.21078/) (2024) - [Exponential convergence rates for momentum stochastic gradient descent in the overparametrized setting](https://mlsanity.com/paper/2302.03550/) (2023) - [Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent](https://mlsanity.com/paper/2302.07125/) (2023) - [Backward error analysis and the qualitative behaviour of stochastic optimization algorithms: Application to stochastic coordinate descent](https://mlsanity.com/paper/2309.02082/) (2023) - [Adam-family Methods with Decoupled Weight Decay in Deep Learning](https://mlsanity.com/paper/2310.08858/) (2023) - [Convergence of stochastic gradient descent under a local Lojasiewicz condition for deep neural networks](https://mlsanity.com/paper/2304.09221/) (2023) - [Stochastic Modified Flows for Riemannian Stochastic Gradient Descent](https://mlsanity.com/paper/2402.03467/) (2024) - [Effective continuous equations for adaptive SGD: a stochastic analysis view](https://mlsanity.com/paper/2509.21614/) (2025) ## Built on by (most cited first, 10 of 10) - [Towards Understanding Adam Convergence on Highly Degenerate Polynomials](https://mlsanity.com/paper/2603.09581/) (2026) - [Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method](https://mlsanity.com/paper/2603.18899/) (2026) - [A Rod Flow Model for Adam at the Edge of Stability](https://mlsanity.com/paper/2605.06821/) (2026) - [Deterministic Adam-Inspired Methods with Accelerated Convergence Rate](https://mlsanity.com/paper/2604.08742/) (2026) - [Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimization](https://mlsanity.com/paper/2605.12878/) (2026) - [Central limit theorem for the averaged Adam optimizer](https://mlsanity.com/paper/2606.21433/) (2026) - [Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks](https://mlsanity.com/paper/2607.04233/) (2026) - [Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Experts](https://mlsanity.com/paper/2601.19811/) (2026) - [The Role of Target Update Frequencies in Q-Learning](https://mlsanity.com/paper/2602.03911/) (2026) - [Strong Averaging Principle and Long-Time Dynamics for Fast-Slow SDEs with Increasing Time-Scale Separation and Degenerate Noise](https://mlsanity.com/paper/2608.23462/) (2026) ## Abstract (arXiv) The Adam optimizer is currently presumably the most popular optimization method in deep learning. In this article we develop an ODE based method to study the Adam optimizer in a fast-slow scaling regime. For fixed momentum parameters and vanishing step-sizes, we show that the Adam algorithm is an asymptotic pseudo-trajectory of the flow of a particular vector field, which is referred to as the Adam vector field. Leveraging properties of asymptotic pseudo-trajectories, we establish convergence results for the Adam algorithm. In particular, in a very general setting we show that if the Adam algorithm converges, then the limit must be a zero of the Adam vector field, rather than a local minimizer or critical point of the objective function. In contrast, in the overparametrized empirical risk minimization setting, the Adam algorithm is able to locally find the set of minima. Specifically, we show that in a neighborhood of the global minima, the objective function serves as a Lyapunov function for the flow induced by the Adam vector field. As a consequence, if the Adam algorithm enters a neighborhood of the global minima infinitely often, it converges to the set of global minima.