# How noise affects the Hessian spectrum in overparameterized neural networks

arXiv: https://arxiv.org/abs/1910.00195 · HTML: https://mlsanity.com/paper/1910.00195/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2019.

## Built on by (most cited first, 17 of 17)

- [Anticorrelated Noise Injection for Improved Generalization](https://mlsanity.com/paper/2202.02831/) (2022)
- [Why flatness does and does not correlate with generalization for deep neural networks](https://mlsanity.com/paper/2103.06219/) (2021)
- [Towards Quantifying the Hessian Structure of Neural Networks](https://mlsanity.com/paper/2505.02809/) (2025)
- [PGrad: Learning Principal Gradients For Domain Generalization](https://mlsanity.com/paper/2305.01134/) (2023)
- [FOCUS: First Order Concentrated Updating Scheme](https://mlsanity.com/paper/2501.12243/) (2025)
- [Stochastic Gradient Descent-Induced Drift of Representation in a Two-Layer Neural Network](https://mlsanity.com/paper/2302.02563/) (2023)
- [On the Theoretical Properties of Noise Correlation in Stochastic Optimization](https://mlsanity.com/paper/2209.09162/) (2022)
- [Flatter, faster: scaling momentum for optimal speedup of SGD](https://mlsanity.com/paper/2210.16400/) (2022)
- [Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules](https://mlsanity.com/paper/2206.00823/) (2022)
- [On the curvature of the loss landscape](https://mlsanity.com/paper/2307.04719/) (2023)
- [Variational Learning Finds Flatter Solutions at the Edge of Stability](https://mlsanity.com/paper/2506.12903/) (2025)
- [Entropic Confinement and Mode Connectivity in Overparameterized Neural Networks](https://mlsanity.com/paper/2512.06297/) (2025)
- [Noise-Driven Exploration and Transient Freezing Select Flat Minima in Stochastic Gradient Descent](https://mlsanity.com/paper/2601.10962/) (2026)
- [Wolkowicz-Styan Upper Bound on the Hessian Eigenspectrum for Cross-Entropy Loss in Nonlinear Smooth Neural Networks](https://mlsanity.com/paper/2604.10202/) (2026)
- [Stochastic Thermodynamics of Learning Parametric Probabilistic Models](https://mlsanity.com/paper/2310.19802/) (2023)
- [Neural network optimization strategies and the topography of the loss landscape](https://mlsanity.com/paper/2602.21276/) (2026)
- [Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks](https://mlsanity.com/paper/2606.28662/) (2026)

## Abstract (arXiv)

Stochastic gradient descent (SGD) forms the core optimization method for deep neural networks. While some theoretical progress has been made, it still remains unclear why SGD leads the learning dynamics in overparameterized networks to solutions that generalize well. Here we show that for overparameterized networks with a degenerate valley in their loss landscape, SGD on average decreases the trace of the Hessian of the loss. We also generalize this result to other noise structures and show that isotropic noise in the non-degenerate subspace of the Hessian decreases its determinant. In addition to explaining SGDs role in sculpting the Hessian spectrum, this opens the door to new optimization approaches that may confer better generalization performance. We test our results with experiments on toy models and deep neural networks.