# The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size

arXiv: https://arxiv.org/abs/1811.07062 · HTML: https://mlsanity.com/paper/1811.07062/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2018.

## Built on by (most cited first, 13 of 13)

- [The Persistence of Neural Collapse Despite Low-Rank Bias](https://mlsanity.com/paper/2410.23169/) (2024)
- [Unifying Low Dimensional Spectra in Deep Learning](https://mlsanity.com/paper/2404.06106/) (2024)
- [Exact Gauss-Newton Optimization for Training Deep Neural Networks](https://mlsanity.com/paper/2405.14402/) (2024)
- [Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching](https://mlsanity.com/paper/2306.07960/) (2023)
- [DOME: Improving Signal-to-Noise in Stochastic Gradient Descent via Sharp-Direction Subspace Filtering](https://mlsanity.com/paper/2507.03545/) (2025)
- [MIO : Mutual Information Optimization using Self-Supervised Binary Contrastive Learning](https://mlsanity.com/paper/2111.12664/) (2021)
- [The Geometric Anatomy of Capability Acquisition in Transformers](https://mlsanity.com/paper/2602.15997/) (2026)
- [Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks](https://mlsanity.com/paper/2201.07934/) (2022)
- [Second-Order, First-Class: A Composable Stack for Curvature-Aware Training](https://mlsanity.com/paper/2603.25976/) (2026)
- [Fast Gauss-Newton for Multiclass Cross-Entropy](https://mlsanity.com/paper/2605.06081/) (2026)
- [Thermodynamic Signatures of Reasoning: Free-Energy and Spectral-Form-Factor Diagnostics for Hallucination Detection in Large Language Models](https://mlsanity.com/paper/2606.19404/) (2026)
- [Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking](https://mlsanity.com/paper/2610.09919/) (2026)
- [Fisher information and the geometry of memorization in neural networks](https://mlsanity.com/paper/2610.16161/) (2026)

## Abstract (arXiv)

We apply state-of-the-art tools in modern high-dimensional numerical linear algebra to approximate efficiently the spectrum of the Hessian of modern deepnets, with tens of millions of parameters, trained on real data. Our results corroborate previous findings, based on small-scale networks, that the Hessian exhibits "spiked" behavior, with several outliers isolated from a continuous bulk. We decompose the Hessian into different components and study the dynamics with training and sample size of each term individually.