mlsanityData to 7 Oct 2026

Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians

arXiv 1901.08244 ↗First posted 24 Jan 2019cs.LG stat.MLcited by 68

Not on the map. Why?

Abstract (arXiv)

We consider deep classifying neural networks. We expose a structure in the derivative of the logits with respect to the parameters of the model, which is used to explain the existence of outliers in the spectrum of the Hessian. Previous works decomposed the Hessian into two components, attributing the outliers to one of them, the so-called Covariance of gradients. We show this term is not a Covariance but a second moment matrix, i.e., it is influenced by means of gradients. These means possess an additive two-way structure that is the source of the outliers in the spectrum. This structure can be used to approximate the principal subspace of the Hessian using certain "averaging" operations, avoiding the need for high-dimensional eigenanalysis. We corroborate this claim across different datasets, architectures and sample sizes.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 8; 2023: 11; 2024: 13; 2025: 8; 2026: 16.

68 papers.

Sort:

  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  2. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  3. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  4. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  5. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  6. 2024A New Perspective on Shampoo's Preconditionercited by 54
  7. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  8. 2024Does SGD really happen in tiny subspaces?cited by 35
  9. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  10. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  11. 2021Power-law escape rate of SGDcited by 29
  12. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
38 more
  1. 2021How many degrees of freedom do we need to train deep networks: a loss landscape perspectivecited by 28
  2. 2022Implicit regularization of dropoutcited by 24
  3. 2024Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networkscited by 21
  4. 2023Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networkscited by 20
  5. 2023A Rainbow in Deep Network Black Boxescited by 18
  6. 2021ViViT: Curvature access through the generalized Gauss-Newton's low-rank structurecited by 16
  7. 2022On the Power-Law Hessian Spectrums in Deep Learningcited by 14
  8. 2023Differentially Private Sharpness-Aware Trainingcited by 14
  9. 2021Does the Data Induce Capacity Control in Deep Learning?cited by 13
  10. 2022On the Overlooked Structure of Stochastic Gradientscited by 13
  11. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  12. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  13. 2021Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stoppingcited by 12
  14. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 12
  15. 2023Stochastic Modified Equations and Dynamics of Dropout Algorithmcited by 11
  16. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  17. 2025Position: Curvature Matrices Should Be Democratized via Linear Operatorscited by 9
  18. 2021Appearance of Random Matrix Theory in Deep Learningcited by 7
  19. 2023On progressive sharpening, flat minima and generalisationcited by 6
  20. 2026A Defense of the Quadratic Modelcited by 6
  21. 2023Unveiling the Hessian's Connection to the Decision Boundarycited by 5
  22. 2025Compute-Optimal LLMs Provably Generalize Better With Scalecited by 5
  23. 2024Deconstructing the Goldilocks Zone of Neural Network Initializationcited by 4
  24. 2023The instabilities of large learning rate training: a loss landscape viewcited by 3
  25. 2026Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimizationcited by 3
  26. 2026AMUSE: Anytime Muon with Stable Gradient Evaluationcited by 3
  27. 2026First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformerscited by 3
  28. 2021Dynamic Game Theoretic Neural Optimizercited by 2
  29. 2021Dropout in Training Neural Networks: Flatness of Solution and Noise Structurecited by 2
  30. 2026On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvaturecited by 2
  31. 2026Distributional Spectral Diagnostics for Localizing Grokking Transitionscited by 2
  32. 2021Structured Directional Pruning via Perturbation Orthogonal Projectioncited by 1
  33. 2022There is a Singularity in the Loss Landscapecited by 1
  34. 2023FOSI: Hybrid First and Second Order Optimizationcited by 1
  35. 2024Training on the Edge of Stability Is Caused by Layerwise Jacobian Alignmentcited by 1
  36. 2024Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapescited by 1
  37. 2025Connecting Parameter Magnitudes and Hessian Eigenspaces at Scale using Sketched Methodscited by 1
  38. 2025Accelerating Neural Network Training Along Sharp and Flat Directionscited by 1

50 most cited of 68 All 68 (JSON)