# Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails

arXiv: https://arxiv.org/abs/2603.03099 · HTML: https://mlsanity.com/paper/2603.03099/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2026.

## Built on by (most cited first, 4 of 4)

- [When Models Learn to Ask Why: Adaptive Causal Reasoning for Trustworthy Medical Vision-Language Models](https://mlsanity.com/paper/2603.23085/) (2026)
- [Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization](https://mlsanity.com/paper/2605.04269/) (2026)
- [PowerStep: Memory-Efficient Adaptive Optimization via ℓp-Norm Steepest Descent](https://mlsanity.com/paper/2605.10335/) (2026)
- [Adam under Generalized Smoothness with Second-Moment-Type Stochastic Gradients](https://mlsanity.com/paper/2609.37787/) (2026)

## Abstract (arXiv)

Despite Adam demonstrating faster empirical convergence than SGD in many applications, much of the existing theory yields guarantees essentially comparable to those of SGD, leaving the empirical performance gap insufficiently explained. In this paper, we uncover a key second-moment normalization in Adam and develop a stopping-time/martingale analysis that provably distinguishes Adam from SGD under the classical bounded variance model (a second moment assumption). In particular, we establish the first theoretical separation between the high-probability convergence behaviors of the two methods: Adam achieves a δ-1/2 dependence on the confidence parameter δ, whereas corresponding high-probability guarantee for SGD necessarily incurs at least a δ-1 dependence.