Deterministic Adam-Inspired Methods with Accelerated Convergence Rate
Not on the map. Why?
Abstract (arXiv)
Adam is widely used, but its convergence theory remains incomplete even in the deterministic full-batch setting because momentum and adaptive preconditioning are tightly coupled. For smooth convex objectives, we split the momentum variable through variable-and-operator splitting, which reveals the acceleration mechanism. We then combine a Hessian-driven correction with Adam-style feedback based on the gradient magnitude. The resulting Adam-HNAG (Hessian-driven Nesterov accelerated gradient with Adam-style adaptive preconditioning) flow admits a nonnegative energy that decays exponentially. Its discretization yields two methods, Adam-HNAG and the synchronous variant Adam-HNAG-s. Under the stated trajectory-bound and consistency conditions, both methods satisfy a discrete Lyapunov contraction. If the exact adaptive steps are accepted, this contraction gives an O(k-2) objective-value bound. Numerical experiments illustrate their behavior. These results apply to the proposed methods, not to the original Adam recursion.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 28 of its 35 references that are in the map. First posted on arXiv 9 Apr 2026.
Builds on
28 of 35 references.
References · 28
- 2019On the Convergence of Adam and Beyondcited by 841
- 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
- 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
- 2020A Simple Convergence Proof of Adam and Adagradcited by 171
- 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
- 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
- 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
- 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
- 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
- 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 49
- 2018Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov accelerationcited by 39
- 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 33
16 more
- 2018A general system of differential equations to model first order adaptive algorithmscited by 28
- 2019First-order optimization algorithms via inertial systems with Hessian driven dampingcited by 22
- 2018Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning ratecited by 16
- 2025SGD with Adaptive Preconditioning: Unified Analysis and Momentum Accelerationcited by 12
- 2025ODE approximation for the Adam algorithm: General and overparametrized settingcited by 10
- 2022Non-Convergence and Limit Cycles in the Adam optimizercited by 9
- 2025Sharp higher order convergence rates for the Adam optimizercited by 8
- 2024Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equationscited by 7
- 2019First order optimization methods based on Hessian-driven Nesterov accelerated gradient flowcited by 6
- 2021Local Convergence of Adaptive Gradient Descent Optimizerscited by 5
- 2021A Unified Convergence Analysis of First Order Convex Optimization Methods via Strong Lyapunov Functionscited by 5
- 2024Continuous-Time Analysis of Adaptive Optimization and Normalizationcited by 4
- 2026SHANG++: Robust Stochastic Acceleration under Multiplicative Noisecited by 3
- 2025Accelerated Gradient Methods Through Variable and Operator Splittingcited by 2
- 2025HNAG++: An Accelerated Gradient Method with a Refined Asymptotic Rate for Strongly Convex Optimizationcited by 1
- 2026From Adam to Adam-Like Lagrangians: Second-Order Nonlocal Dynamicscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.