# SHANG++: Robust Stochastic Acceleration under Multiplicative Noise

arXiv: https://arxiv.org/abs/2603.09355 · HTML: https://mlsanity.com/paper/2603.09355/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://mlsanity.com/paper/1905.11001/) (2019)
- [Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptron](https://mlsanity.com/paper/1810.07288/) (2018)
- [Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints](https://mlsanity.com/paper/1408.3595/) (2014)
- [The alignment property of SGD noise and how it helps select flat minima: A stability analysis](https://mlsanity.com/paper/2207.02628/) (2022)
- [On the insufficiency of existing momentum schemes for Stochastic Optimization](https://mlsanity.com/paper/1803.05591/) (2018)
- [Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network](https://mlsanity.com/paper/1902.07111/) (2019)
- [On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settings](https://mlsanity.com/paper/2002.12414/) (2020)
- [Multiplicative noise and heavy tails in stochastic optimization](https://mlsanity.com/paper/2006.06293/) (2020)
- [A Continuized View on Nesterov Acceleration for Stochastic Gradient Descent and Randomized Gossip](https://mlsanity.com/paper/2106.07644/) (2021)
- [Nesterov's method with decreasing learning rate leads to accelerated stochastic gradient descent](https://mlsanity.com/paper/1908.07861/) (2019)
- [On the fast convergence of minibatch heavy ball momentum](https://mlsanity.com/paper/2206.07553/) (2022)
- [Provable non-accelerations of the heavy-ball method](https://mlsanity.com/paper/2307.11291/) (2023)
- [Nesterov acceleration despite very noisy gradients](https://mlsanity.com/paper/2302.05515/) (2023)
- [First order optimization methods based on Hessian-driven Nesterov accelerated gradient flow](https://mlsanity.com/paper/1912.09276/) (2019)
- [A Unified Convergence Analysis of First Order Convex Optimization Methods via Strong Lyapunov Functions](https://mlsanity.com/paper/2108.00132/) (2021)
- [A Diffusion Approximation Theory of Momentum SGD in Nonconvex Optimization](https://mlsanity.com/paper/1802.05155/) (2018)
- [Gradient correlation is a key ingredient to accelerate SGD with momentum](https://mlsanity.com/paper/2410.07870/) (2024)

## Built on by (most cited first, 3 of 3)

- [Deterministic Adam-Inspired Methods with Accelerated Convergence Rate](https://mlsanity.com/paper/2604.08742/) (2026)
- [Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimization](https://mlsanity.com/paper/2605.12878/) (2026)
- [Geometric Moment Contraction for Stochastic Nesterov Acceleration](https://mlsanity.com/paper/2609.31303/) (2026)

## Abstract (arXiv)

Under the multiplicative noise scaling (MNS) condition, original Nesterov acceleration is provably sensitive to noise and may diverge when gradient noise overwhelms the signal. In this paper, we develop two accelerated stochastic gradient descent methods by discretizing the Hessian-driven Nesterov accelerated gradient flow. We first derive SHANG, a direct semi-implicit discretization that already improves stability under MNS. We then introduce SHANG++, which adds a damping correction and achieves faster convergence with greater noise robustness. We establish convergence guarantees for both convex and strongly convex objectives under MNS, together with explicit parameter choices. In our experiments, SHANG++ performs consistently well across convex problems and applications in deep learning. In a dedicated noise experiment on ResNet-34, a single hyperparameter configuration maintains accuracy within one percentage point of the noise-free setting. Across all experiments, SHANG++ outperforms existing accelerated methods in robustness and efficiency, with minimal parameter sensitivity.