mlsanityData to 7 Oct 2026

Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks

arXiv 2010.04261 ↗First posted 8 Oct 2020cs.LG cs.NE stat.MLcited by 45

Not on the map. Why?

Abstract (arXiv)

Hessian captures important properties of the deep neural network loss landscape. Previous works have observed low rank structure in the Hessians of neural networks. In this paper, we propose a decoupling conjecture that decomposes the layer-wise Hessians of a network as the Kronecker product of two smaller matrices. We can analyze the properties of these smaller matrices and prove the structure of top eigenspace random 2-layer networks. The decoupling conjecture has several other interesting implications - top eigenspaces for different models have surprisingly high overlap, and top eigenvectors form low rank matrices when they are reshaped into the same shape as the corresponding weight matrix. All of these can be verified empirically for deeper networks. Finally, we use the structure of layer-wise Hessian to get better explicit generalization bounds for neural networks.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 6; 2023: 6; 2024: 10; 2025: 10; 2026: 8.

45 papers.

Sort:

  1. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  2. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  3. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  4. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  5. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  6. 2022SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximationcited by 51
  7. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  8. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  9. 2023Symmetry Induces Structure and Constraint of Learningcited by 25
  10. 2023ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimizationcited by 24
  11. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  12. 2023Noise Stability Optimization for Finding Flat Minima: A Hessian-based Regularization Approachcited by 14
33 more
  1. 2021Does the Data Induce Capacity Control in Deep Learning?cited by 13
  2. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  3. 2024Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimizationcited by 12
  4. 2021Second-Order Neural ODE Optimizercited by 11
  5. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  6. 2026RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimizationcited by 11
  7. 2023The Hessian perspective into the Nature of Convolutional Neural Networkscited by 10
  8. 2021Kronecker-factored Quasi-Newton Methods for Deep Learningcited by 6
  9. 2022Federated Optimization of Smooth Loss Functionscited by 6
  10. 2024Theoretical characterisation of the Gauss-Newton conditioning in Neural Networkscited by 5
  11. 2025Spectral Collapse Drives Loss of Plasticity in Deep Continual Learningcited by 5
  12. 2023Learning Capacity: A Measure of the Effective Dimensionality of a Modelcited by 4
  13. 2022TorchNTK: A Library for Calculation of Neural Tangent Kernels of PyTorch Modelscited by 3
  14. 2025Optimal Brain Apoptosiscited by 3
  15. 2025Converge Faster, Talk Less: Hessian-Informed Federated Zeroth-Order Optimizationcited by 3
  16. 2026Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimizationcited by 3
  17. 2021Dynamic Game Theoretic Neural Optimizercited by 2
  18. 2022Gradient Descent for Low-Rank Functionscited by 2
  19. 2025Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scalecited by 2
  20. 2026LoRIF: Low-Rank Influence Functions for Scalable Training Data Attributioncited by 2
  21. 2024Improving Knowledge Distillation in Transfer Learning with Layer-wise Learning Ratescited by 1
  22. 2024Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapescited by 1
  23. 2025On the Convergence Analysis of Muoncited by 1
  24. 2025Minimizing Layerwise Activation Norm Improves Generalization in Federated Learningcited by 1
  25. 2026Wolkowicz-Styan Upper Bound on the Hessian Eigenspectrum for Cross-Entropy Loss in Nonlinear Smooth Neural Networkscited by 1
  26. 2024Q-Newton: Hybrid Quantum-Classical Scheduling for Accelerating Neural Network Training with Newton's Gradient Descentcited by 0
  27. 2024Loss Gradient Gaussian Width based Generalization and Optimization Guaranteescited by 0
  28. 2025Bandit Guided Submodular Curriculum for Adaptive Subset Selectioncited by 0
  29. 2025An Equivariance Toolbox for Learning Dynamicscited by 0
  30. 2026Curvature-Aligned Probing for Local Loss-Landscape Stabilizationcited by 0
  31. 2026Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponentcited by 0
  32. 2026Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networkscited by 0
  33. 2026Functional Equivalence and Geometric Diversity in Neural Network Approximations: An Empirical Characterizationcited by 0