mlsanityData to 14 Oct 2026

Towards Quantifying the Hessian Structure of Neural Networks

arXiv 2505.02809 ↗First posted 5 May 2025cs.LG math.OC stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

Empirical studies reported that the Hessian matrix of neural networks (NNs) exhibits a near-block-diagonal structure, yet its theoretical foundation remains unclear. In this work, we reveal that the reported Hessian structure comes from a mixture of two forces: a ``static force'' rooted in the architecture design, and a ''dynamic force'' arisen from training. We then provide a rigorous theoretical analysis of ''static force'' at random initialization. We study linear models and 1-hidden-layer networks for classification tasks with C classes. By leveraging random matrix theory, we compare the limit distributions of the diagonal and off-diagonal Hessian blocks and find that the block-diagonal structure arises as C becomes large. Our findings reveal that C is one primary driver of the near-block-diagonal structure. These results may shed new light on the Hessian structure of large language models (LLMs), which typically operate with a large C exceeding 104.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 62 of its 89 references that are in the map. First posted on arXiv 5 May 2025.

Builds on

62 of 89 references.

References · 62

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  5. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  6. 2025Muon is Scalable for LLM Trainingcited by 428
  7. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  8. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  9. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  10. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  11. 2022How Do Vision Transformers Work?cited by 307
  12. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
38 more
  1. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  2. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  3. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
  4. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  5. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  6. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  7. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
  8. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 159
  9. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  10. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 146
  11. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  12. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  13. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  14. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
  15. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  16. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  17. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  18. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 88
  19. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  20. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  21. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  22. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  23. 2019Optimization for deep learning: theory and algorithmscited by 70
  24. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  25. 2020Practical Quasi-Newton Methods for Training Deep Neural Networkscited by 68
  26. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
  27. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  28. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  29. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  30. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  31. 2019Emergent properties of the local geometry of neural loss landscapescited by 46
  32. 2020Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networkscited by 45
  33. 2015Natural Neural Networkscited by 44
  34. 2020A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularizationcited by 44
  35. 2019Hessian based analysis of SGD for Deep Nets: Dynamics and Generalizationcited by 37
  36. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  37. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 30
  38. 2024Pushing the Limits of Large Language Model Quantization via the Linearity Theoremcited by 21

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 11.

13 papers.

Sort:

  1. 2026RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimizationcited by 15
  2. 2026Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentumcited by 14
  3. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 6
  4. 2025On the Convergence Analysis of Muoncited by 5
  5. 2026Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizercited by 5
  6. 2025Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functionscited by 3
  7. 2026Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMscited by 2
  8. 2026Muse: Representation Geometry of Muon Beyond Normalized Momentumcited by 1
  9. 2026MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioningcited by 1
  10. 2026Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrencecited by 1
  11. 2026One-Sided Matrix Completion from Ultra-Sparse Samplescited by 0
  12. 2026Outer-Momentum Restarting in High-Dimensional Two-Phase Optimizationcited by 0
1 more
  1. 2026Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compressioncited by 0