mlsanityData to 7 Oct 2026

An Investigation into Neural Net Optimization via Hessian Eigenvalue Density

arXiv 1901.10159 ↗First posted 29 Jan 2019cs.LG stat.MLcited by 289

Not on the map. Why?

Abstract (arXiv)

To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process. Using this, we study a number of hypotheses concerning smoothness, curvature, and sharpness in the deep learning literature. We then thoroughly analyze a crucial structural feature of the spectra: in non-batch normalized networks, we observe the rapid appearance of large isolated eigenvalues in the spectrum, along with a surprising concentration of the gradient in the corresponding eigenspaces. In batch normalized networks, these two effects are almost absent. We characterize these effects, and explain how they affect optimization speed through both theory and experiments. As part of this work, we adapt advanced tools from numerical linear algebra that allow scalable and accurate estimation of the entire Hessian spectrum of ImageNet-scale neural networks; this technique may be of independent interest in other applications.

Key facts. Cited by 289 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 31; 2023: 40; 2024: 44; 2025: 48; 2026: 98.

289 papers.

Sort:

By area:

  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  2. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  3. 2022How Do Vision Transformers Work?cited by 305
  4. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  5. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  6. 2023Understanding plasticity in neural networkscited by 170
  7. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  8. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  9. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  10. 2021Scaling Up Influence Functionscited by 130
  11. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  12. 2024Challenges in Training PINNs: A Loss Landscape Perspectivecited by 109
38 more
  1. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 100
  2. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  3. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  4. 2022A Closer Look at Smoothness in Domain Adversarial Trainingcited by 85
  5. 2022When Do Flat Minima Optimizers Work?cited by 78
  6. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  7. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  8. 2024Friendly Sharpness-Aware Minimizationcited by 52
  9. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 51
  10. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  11. 2021A study on the plasticity of neural networkscited by 44
  12. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  13. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 39
  14. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  15. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  16. 2024Does SGD really happen in tiny subspaces?cited by 35
  17. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  18. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 33
  19. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  20. 2024Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?cited by 31
  21. 2021How many degrees of freedom do we need to train deep networks: a loss landscape perspectivecited by 28
  22. 2022Escaping Saddle Points for Effective Generalization on Class-Imbalanced Datacited by 28
  23. 2023Theoretical and Practical Perspectives on what Influence Functions Docited by 28
  24. 2023An operator preconditioning perspective on training in physics-informed machine learningcited by 27
  25. 2022Stochastic Weight Averaging Revisitedcited by 26
  26. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  27. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  28. 2023Symmetry Induces Structure and Constraint of Learningcited by 25
  29. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
  30. 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenoncited by 24
  31. 2023Memorization Through the Lens of Curvature of Loss Function Around Samplescited by 24
  32. 2023Improving Pixel-based MIM by Reducing Wasted Modeling Capabilitycited by 24
  33. 2023ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimizationcited by 24
  34. 2023Improving Generalization of Adversarial Training via Robust Critical Fine-Tuningcited by 23
  35. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  36. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
  37. 2021On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decaycited by 20
  38. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 20

50 most cited of 289 All 289 (JSON)