# Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks

arXiv: https://arxiv.org/abs/2510.11354 · HTML: https://mlsanity.com/paper/2510.11354/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2025.

## Abstract (arXiv)

Adam is a popular and widely used adaptive gradient method in deep learning, which has also received tremendous focus in theoretical research. However, most existing theoretical work primarily analyzes its full-batch version, which differs fundamentally from the stochastic variant used in practice. Unlike SGD, stochastic Adam does not converge to its full-batch counterpart even with infinitesimal learning rates. We present the first theoretical characterization of how batch size affects Adam's generalization, analyzing two-layer over-parameterized CNNs on image data. Our results reveal that while both Adam and AdamW with proper weight decay λ converge to poor test error solutions, their mini-batch variants can achieve near-zero test error. We further prove Adam has a strictly smaller effective weight decay bound than AdamW, theoretically explaining why Adam requires more sensitive λ tuning. Extensive experiments validate our findings, demonstrating the critical role of batch size and weight decay in Adam's generalization performance.