# Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimization

arXiv: https://arxiv.org/abs/2605.12878 · HTML: https://mlsanity.com/paper/2605.12878/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 13 May 2026.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Shampoo: Preconditioned Stochastic Tensor Optimization](https://mlsanity.com/paper/1802.09568/) (2018)
- [On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization](https://mlsanity.com/paper/1808.02941/) (2018)
- [On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization](https://mlsanity.com/paper/1808.05671/) (2018)
- [Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimization](https://mlsanity.com/paper/1810.02263/) (2018)
- [Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration](https://mlsanity.com/paper/1807.06766/) (2018)
- [A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms](https://mlsanity.com/paper/2009.06125/) (2020)
- [A general system of differential equations to model first order adaptive algorithms](https://mlsanity.com/paper/1810.13108/) (2018)
- [AdaGrad under Anisotropic Smoothness](https://mlsanity.com/paper/2406.15244/) (2024)
- [ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Rate](https://mlsanity.com/paper/2411.02853/) (2024)
- [Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate](https://mlsanity.com/paper/1805.07557/) (2018)
- [Convergence rates for the Adam optimizer](https://mlsanity.com/paper/2407.21078/) (2024)
- [SGD with Adaptive Preconditioning: Unified Analysis and Momentum Acceleration](https://mlsanity.com/paper/2506.23803/) (2025)
- [ODE approximation for the Adam algorithm: General and overparametrized setting](https://mlsanity.com/paper/2511.04622/) (2025)
- [Sharp higher order convergence rates for the Adam optimizer](https://mlsanity.com/paper/2504.19426/) (2025)
- [Nesterov acceleration despite very noisy gradients](https://mlsanity.com/paper/2302.05515/) (2023)
- [Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equations](https://mlsanity.com/paper/2411.09734/) (2024)
- [First order optimization methods based on Hessian-driven Nesterov accelerated gradient flow](https://mlsanity.com/paper/1912.09276/) (2019)
- [Convergence of adaptive algorithms for weakly convex constrained optimization](https://mlsanity.com/paper/2006.06650/) (2020)
- [Improving the Adaptive Moment Estimation (ADAM) stochastic optimizer through an Implicit-Explicit (IMEX) time-stepping approach](https://mlsanity.com/paper/2403.13704/) (2024)
- [SHANG++: Robust Stochastic Acceleration under Multiplicative Noise](https://mlsanity.com/paper/2603.09355/) (2026)
- [UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization](https://mlsanity.com/paper/2305.05675/) (2023)
- [Accelerated Gradient Methods Through Variable and Operator Splitting](https://mlsanity.com/paper/2505.04065/) (2025)
- [Incorporating Preconditioning into Accelerated Approaches: Theoretical Guarantees and Practical Improvement](https://mlsanity.com/paper/2505.23510/) (2025)
- [Deterministic Adam-Inspired Methods with Accelerated Convergence Rate](https://mlsanity.com/paper/2604.08742/) (2026)

## Built on by (most cited first, 1 of 1)

- [Convergence rates for the RMSprop optimizer with full control of the hyperparameters](https://mlsanity.com/paper/2608.30382/) (2026)

## Abstract (arXiv)

We propose Adam-SHANG, a Lyapunov-guided Adam-type method that couples momentum, adaptive preconditioning, and a curvature-aware correction through a more stable lagged-preconditioner update. For stochastic smooth convex optimization, we prove convergence in expectation under an admissible stepsize condition that can always be satisfied by a conservative spectral bound, without imposing global monotonicity on the second-moment sequence. To obtain a less conservative practical rule, we introduce a computable trace-ratio stepsize, motivated by a local coordinatewise alignment condition. The same structural update is also tested beyond the convex setting with simplified parameters. Experiments validate the predicted stochastic decay and show competitive training performance against Adam and AdamW on deep learning tasks.