Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations
Not on the map. Why?
Abstract (arXiv)
During SGD training, the gradients often align strongly with the dominant subspace spanned by the top-k eigenvectors of the Hessian of the loss. While this seems to naturally imply that loss reduction mainly occurs within this space, prior work has shown that updates within this dominant subspace make no meaningful progress in reducing the loss. In this work, we argue that the dominant subspace is better understood not as the main space for loss reduction, but as a key subspace for explaining the sharpness dynamics of mini-batch SGD. To explain the role of the dominant subspace in reducing top-k sharpness, we show how the averaged gradient over fluctuations in the dominant directions produces a sharpness correction term, and derive a sharpness correction term induced by mini-batch noise in the dominant directions. Experimental results show that adding the derived correction term to GD brings the sharpness evolution of GD closer to that of SGD.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 25 Jul 2026.
Builds on
27 of 37 references.
References · 27
- 2017Attention Is All You Needcited by 59,932
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2017Sharp Minima Can Generalize For Deep Netscited by 462
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
- 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
- 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
- 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
- 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
- 2017Three Factors Influencing Minima in SGDcited by 270
- 2018Gradient Descent Happens in a Tiny Subspacecited by 208
- 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
15 more
- 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
- 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
- 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
- 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
- 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
- 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
- 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
- 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
- 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
- 2024Understanding Optimization in Deep Learning with Central Flowscited by 57
- 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
- 2019On the interplay between noise and curvature and its effect on optimization and generalizationcited by 44
- 2024Does SGD really happen in tiny subspaces?cited by 35
- 2026Suspicious Alignment of SGD: A Fine-Grained Step Size Condition Analysiscited by 5
- 2025Accelerating Neural Network Training Along Sharp and Flat Directionscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.