mlsanityData to 7 Oct 2026

On the Power-Law Hessian Spectrums in Deep Learning

arXiv 2201.13011 ↗First posted 31 Jan 2022cs.LG physics.bio-ph q-bio.BMcited by 14

Not on the map. Why?

Abstract (arXiv)

It is well-known that the Hessian of deep loss landscape matters to optimization, generalization, and even robustness of deep learning. Recent works empirically discovered that the Hessian spectrum in deep learning has a two-component structure that consists of a small number of large eigenvalues and a large number of nearly-zero eigenvalues. However, the theoretical mechanism or the mathematical behind the Hessian spectrum is still largely under-explored. To the best of our knowledge, we are the first to demonstrate that the Hessian spectrums of well-trained deep neural networks exhibit simple power-law structures. Inspired by the statistical physical theories and the spectral analysis of natural proteins, we provide a maximum-entropy theoretical interpretation for explaining why the power-law structure exist and suggest a spectral parallel between protein evolution and training of deep neural networks. By conducing extensive experiments, we further use the power-law spectral framework as a useful tool to explore multiple novel behaviors of deep learning.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 44 of its 98 references that are in the map. First posted on arXiv 31 Jan 2022.

Builds on

44 of 98 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  5. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  6. 2019On the Convergence of Adam and Beyondcited by 834
  7. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  8. 2017Exploring Generalization in Deep Learningcited by 595
  9. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  10. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  11. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  12. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
32 more
  1. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  2. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  3. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  4. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  5. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  6. 2017Three Factors Influencing Minima in SGDcited by 270
  7. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  8. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  9. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 181
  10. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  11. 2020Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustnesscited by 130
  12. 2020The Heavy-Tail Phenomenon in SGDcited by 122
  13. 2020A Survey of Label-noise Representation Learning: Past, Present and Futurecited by 121
  14. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 116
  15. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  16. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  17. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  18. 2018The Description Length of Deep Learning Modelscited by 71
  19. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  20. 2019Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysiscited by 58
  21. 2019Emergent properties of the local geometry of neural loss landscapescited by 46
  22. 2019On the interplay between noise and curvature and its effect on optimization and generalizationcited by 44
  23. 2020A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularizationcited by 44
  24. 2020Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentumcited by 40
  25. 2019Hessian based analysis of SGD for Deep Nets: Dynamics and Generalizationcited by 37
  26. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  27. 2018The Goldilocks zone: Towards better understanding of neural network loss landscapescited by 32
  28. 2019diffGrad: An Optimization Method for Convolutional Neural Networkscited by 29
  29. 2021Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalizationcited by 28
  30. 2019Non-Gaussianity of Stochastic Gradient Noisecited by 22
  31. 2019The asymptotic spectrum of the Hessian of DNN throughout trainingcited by 19
  32. 2016Learning Thermodynamics with Boltzmann Machinescited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 4; 2026: 4.

14 papers.

Sort:

By area:

  1. 2023Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Trainingcited by 19
  2. 2025Uncertainty Quantification with the Empirical Neural Tangent Kernelcited by 11
  3. 2025Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learningcited by 9
  4. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 9
  5. 2025HessFormer: Hessians at Foundation Scalecited by 4
  6. 2024A Precise Characterization of SGD Stability Using Loss Surface Geometrycited by 2
  7. 2024Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Featurescited by 1
  8. 2024Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapescited by 1
  9. 2024Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysiscited by 1
  10. 2026Wolkowicz-Styan Upper Bound on the Hessian Eigenspectrum for Cross-Entropy Loss in Nonlinear Smooth Neural Networkscited by 1
  11. 2024Loss Gradient Gaussian Width based Generalization and Optimization Guaranteescited by 0
  12. 2026Importance-Guided Basis Selection for Low-Rank Decomposition of Large Language Modelscited by 0
2 more
  1. 2026Fast Gauss-Newton for Multiclass Cross-Entropycited by 0
  2. 2026Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networkscited by 0