mlsanityData to 14 Oct 2026

Improving the Adaptive Moment Estimation (ADAM) stochastic optimizer through an Implicit-Explicit (IMEX) time-stepping approach

arXiv 2403.13704 ↗First posted 20 Mar 2024cs.CE cs.LG cs.NAcited by 4

Not on the map. Why?

Abstract (arXiv)

The Adam optimizer, often used in Machine Learning for neural network training, corresponds to an underlying ordinary differential equation (ODE) in the limit of very small learning rates. This work shows that the classical Adam algorithm is a first-order implicit-explicit (IMEX) Euler discretization of the underlying ODE. Employing the time discretization point of view, we propose new extensions of the Adam scheme obtained by using higher-order IMEX methods to solve the ODE. Based on this approach, we derive a new optimization algorithm for neural network training that performs better than classical Adam on several regression and classification problems.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 11 of its 42 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

11 of 42 references.

References · 11

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2019On the Convergence of Adam and Beyondcited by 841
  5. 2015Cyclical Learning Rates for Training Neural Networkscited by 629
  6. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 206
  7. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  8. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 103
  9. 2020Learning Variational Data Assimilation Models and Solverscited by 36
  10. 2018A general system of differential equations to model first order adaptive algorithmscited by 28
  11. 2023AI-Lorenz: A physics-data-driven framework for black-box and gray-box identification of chaotic systems with symbolic regressioncited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Deep Operator Networks for Bayesian Parameter Estimation in PDEscited by 6
  2. 2026Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimizationcited by 1
  3. 2025Ensemble based Closed-Loop Optimal Control using Physics-Informed Neural Networkscited by 0
  4. 2026Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizerscited by 0