mlsanityData to 14 Oct 2026

Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimization

arXiv 2605.12878 ↗First posted 13 May 2026math.OC cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We propose Adam-SHANG, a Lyapunov-guided Adam-type method that couples momentum, adaptive preconditioning, and a curvature-aware correction through a more stable lagged-preconditioner update. For stochastic smooth convex optimization, we prove convergence in expectation under an admissible stepsize condition that can always be satisfied by a conservative spectral bound, without imposing global monotonicity on the second-moment sequence. To obtain a less conservative practical rule, we introduce a computable trace-ratio stepsize, motivated by a local coordinatewise alignment condition. The same structural update is also tested beyond the convex setting with simplified parameters. Experiments validate the predicted stochastic decay and show competitive training performance against Adam and AdamW on deep learning tasks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

26 of 36 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019On the Convergence of Adam and Beyondcited by 841
  3. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  4. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  5. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  6. 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 49
  7. 2018Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov accelerationcited by 39
  8. 2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 33
  9. 2018A general system of differential equations to model first order adaptive algorithmscited by 28
  10. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  11. 2024ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Ratecited by 18
  12. 2018Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning ratecited by 16
14 more
  1. 2024Convergence rates for the Adam optimizercited by 15
  2. 2025SGD with Adaptive Preconditioning: Unified Analysis and Momentum Accelerationcited by 12
  3. 2025ODE approximation for the Adam algorithm: General and overparametrized settingcited by 10
  4. 2025Sharp higher order convergence rates for the Adam optimizercited by 8
  5. 2023Nesterov acceleration despite very noisy gradientscited by 7
  6. 2024Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equationscited by 7
  7. 2019First order optimization methods based on Hessian-driven Nesterov accelerated gradient flowcited by 6
  8. 2020Convergence of adaptive algorithms for weakly convex constrained optimizationcited by 6
  9. 2024Improving the Adaptive Moment Estimation (ADAM) stochastic optimizer through an Implicit-Explicit (IMEX) time-stepping approachcited by 4
  10. 2026SHANG++: Robust Stochastic Acceleration under Multiplicative Noisecited by 3
  11. 2023UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimizationcited by 2
  12. 2025Accelerated Gradient Methods Through Variable and Operator Splittingcited by 2
  13. 2025Incorporating Preconditioning into Accelerated Approaches: Theoretical Guarantees and Practical Improvementcited by 2
  14. 2026Deterministic Adam-Inspired Methods with Accelerated Convergence Ratecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Convergence rates for the RMSprop optimizer with full control of the hyperparameterscited by 0