mlsanityData to 14 Oct 2026

Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond

arXiv 1611.07476 ↗First posted 22 Nov 2016cs.LGcited by 159

Not on the map. Why?

Abstract (arXiv)

We look at the eigenvalues of the Hessian of a loss function before and after training. The eigenvalue distribution is seen to be composed of two parts, the bulk which is concentrated around zero, and the edges which are scattered away from zero. We present empirical evidence for the bulk indicating how over-parametrized the system is, and for the edges that depend on the input data.

Key facts. Cited by 159 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 18; 2022: 20; 2023: 26; 2024: 28; 2025: 31; 2026: 36.

159 papers.

Sort:

By area:

  1. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  2. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  3. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  4. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  5. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  6. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
  7. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 58
  8. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  9. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 53
  10. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  11. 2024Does SGD really happen in tiny subspaces?cited by 36
  12. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 35
38 more
  1. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  2. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  3. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  4. 2025Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Homecited by 33
  5. 2021How many degrees of freedom do we need to train deep networks: a loss landscape perspectivecited by 28
  6. 2021Stability & Generalisation of Gradient Descent for Shallow Neural Networks without the Neural Tangent Kernelcited by 26
  7. 2023Symmetry Induces Structure and Constraint of Learningcited by 25
  8. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  9. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  10. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  11. 2023Revisiting the Fragility of Influence Functionscited by 20
  12. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  13. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 19
  14. 2023A Rainbow in Deep Network Black Boxescited by 18
  15. 2023Sketchy: Memory-efficient Adaptive Regularization with Frequent Directionscited by 17
  16. 2026Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioningcited by 17
  17. 2021Learning with Gradient Descent and Weakly Convex Lossescited by 16
  18. 2021ViViT: Curvature access through the generalized Gauss-Newton's low-rank structurecited by 16
  19. 2021Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic Modelscited by 16
  20. 2021Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry IIcited by 16
  21. 2022On the Power-Law Hessian Spectrums in Deep Learningcited by 16
  22. 2022Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?cited by 16
  23. 2021Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrumscited by 15
  24. 2023Riemannian Laplace approximations for Bayesian neural networkscited by 15
  25. 2024FouRA: Fourier Low Rank Adaptationcited by 15
  26. 2026RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimizationcited by 15
  27. 2022Precision Machine Learningcited by 14
  28. 2026A Defense of the Quadratic Modelcited by 14
  29. 2021Why flatness does and does not correlate with generalization for deep neural networkscited by 13
  30. 2022On the Overlooked Structure of Stochastic Gradientscited by 13
  31. 2023Typical and atypical solutions in non-convex neural networks with discrete and continuous weightscited by 13
  32. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  33. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 13
  34. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  35. 2026Insights on Muon from Simple Quadraticscited by 13
  36. 2021Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stoppingcited by 12
  37. 2022Phenomenology of Double Descent in Finite-Width Neural Networkscited by 12
  38. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11

50 most cited of 159 All 159 (JSON)