mlsanityData to 7 Oct 2026

Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equations

arXiv 2411.09734 ↗First posted 14 Nov 2024cs.LG cs.NA math.NAcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a continuous-time formulation for the AdaGrad, RMSProp, and Adam optimization algorithms by modeling them as first-order integro-differential equations. We perform numerical simulations of these equations, along with stability and convergence analyses, to demonstrate their validity as accurate approximations of the original algorithms. Our results indicate a strong agreement between the behavior of the continuous-time models and the discrete implementations, thus providing a new perspective on the theoretical understanding of adaptive optimization methods.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 14 of its 34 references that are in the map. First posted on arXiv 14 Nov 2024.

Builds on

14 of 34 references.

References · 14

  1. 2018Neural Ordinary Differential Equationscited by 3,733
  2. 2021Convex optimizationcited by 2,479
  3. 2016An overview of gradient descent optimization algorithmscited by 1,021
  4. 2019Hamiltonian Neural Networkscited by 702
  5. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  6. 2020Lagrangian Neural Networkscited by 388
  7. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  8. 2018Deep Neural Networks Motivated by Partial Differential Equationscited by 187
  9. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  10. 2016A Variational Perspective on Accelerated Methods in Optimizationcited by 108
  11. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  12. 2019Continuous Time Analysis of Momentum Methodscited by 32
2 more
  1. 2021Noether Networks: Meta-Learning Useful Conserved Quantitiescited by 26
  2. 2016Stochastic Gradient Descent in Continuous Timecited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Deep Probabilistic Modeling of User Behavior for Anomaly Detection via Mixture Density Networkscited by 6
  2. 2026From Adam to Adam-Like Lagrangians: Second-Order Nonlocal Dynamicscited by 1
  3. 2026Deterministic Adam-Inspired Methods with Accelerated Convergence Ratecited by 1
  4. 2026Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimizationcited by 1
  5. 2025Rolling Ball Optimizer: Learning by ironing out loss landscape wrinklescited by 0
  6. 2026Particle-based Generalised Stochastic Optimisationcited by 0