mlsanityData to 7 Oct 2026

Hessian Eigenspectra of More Realistic Nonlinear Models

arXiv 2103.01519 ↗First posted 2 Mar 2021stat.ML cs.LG math.SPcited by 33

Not on the map. Why?

Abstract (arXiv)

Given an optimization problem, the Hessian matrix and its eigenspectrum can be used in many ways, ranging from designing more efficient second-order algorithms to performing model analysis and regression diagnostics. When nonlinear models and non-convex problems are considered, strong simplifying assumptions are often made to make Hessian spectral analysis more tractable. This leads to the question of how relevant the conclusions of such analyses are for more realistic nonlinear models. In this paper, we exploit deterministic equivalent techniques from random matrix theory to make a precise characterization of the Hessian eigenspectra for a broad family of nonlinear models, including models that generalize the classical generalized linear models, without relying on strong simplifying assumptions used previously. We show that, depending on the data properties, the nonlinear response model, and the loss function, the Hessian can have qualitatively different spectral behaviors: of bounded or unbounded support, with single- or multi-bulk, and with isolated eigenvalues on the left- or right-hand side of the bulk. By focusing on such a simple but nontrivial nonlinear model, our analysis takes a step forward to unveil the theoretical origin of many visually striking features observed in more complex machine learning models.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 47 of its 109 references that are in the map. First posted on arXiv 2 Mar 2021.

Builds on

47 of 109 references.

References · 47

  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  2. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  3. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  4. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  5. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  6. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  7. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  8. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  9. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  10. 2014The Loss Surfaces of Multilayer Networkscited by 278
  11. 2016Deep Learning without Poor Local Minimacited by 233
  12. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
35 more
  1. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  2. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  3. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 159
  4. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  5. 2017Non-convex Optimization for Machine Learningcited by 132
  6. 2014Phase Retrieval via Wirtinger Flow: Theory and Algorithmscited by 123
  7. 2020The Heavy-Tail Phenomenon in SGDcited by 122
  8. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 113
  9. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  10. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  11. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  12. 2018The jamming transition as a paradigm to understand the loss landscape of deep neural networkscited by 61
  13. 2013Phase Retrieval using Alternating Minimizationcited by 58
  14. 2015Solving Random Quadratic Systems of Equations Is Nearly as Easy as Solving Linear Systemscited by 50
  15. 2019Emergent properties of the local geometry of neural loss landscapescited by 46
  16. 2020Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networkscited by 45
  17. 2020A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularizationcited by 44
  18. 2017GIANT: Globally Improved Approximate Newton Method for Distributed Optimizationcited by 43
  19. 2017Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Informationcited by 41
  20. 2017Fundamental Limits of Weak Recovery with Applications to Phase Retrievalcited by 39
  21. 2015Simple, Efficient, and Neural Algorithms for Sparse Codingcited by 37
  22. 2019Hessian based analysis of SGD for Deep Nets: Dynamics and Generalizationcited by 37
  23. 2017Second-Order Optimization for Non-Convex Machine Learning: An Empirical Studycited by 35
  24. 2017Phase Transitions of Spectral Initialization for High-Dimensional Nonconvex Estimationcited by 26
  25. 2013Impact of regularization on Spectral Clusteringcited by 25
  26. 2016Sub-sampled Newton Methods with Non-uniform Samplingcited by 21
  27. 2019The asymptotic spectrum of the Hessian of DNN throughout trainingcited by 19
  28. 2018Estimating the Spectral Density of Large Implicit Matricescited by 16
  29. 2015Phase Transitions in Sparse PCAcited by 15
  30. 2020Optimal Combination of Linear and Spectral Estimators for Generalized Linear Modelscited by 15
  31. 2018Trust Region Based Adversarial Attack on Neural Networkscited by 11
  32. 2020Beyond Random Matrix Theory for Deep Networkscited by 11
  33. 2020Sparse Quantized Spectral Clusteringcited by 9
  34. 2018A Geometric Approach of Gradient Descent Algorithms in Linear Neural Networkscited by 8
  35. 2020Average-case Acceleration Through Spectral Density Estimationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 6; 2024: 9; 2025: 9; 2026: 3.

33 papers.

Sort:

  1. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  2. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  3. 2023A Theory on Adam Instability in Large-Scale Machine Learningcited by 54
  4. 2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
  5. 2025Improving the Scaling Laws of Synthetic Data with Deliberate Practicecited by 17
  6. 2022Visualizing high-dimensional loss landscapes with Hessian directionscited by 16
  7. 2022On double-descent in uncertainty quantification in overparametrized modelscited by 15
  8. 2022On the Power-Law Hessian Spectrums in Deep Learningcited by 14
  9. 2022On the Overlooked Structure of Stochastic Gradientscited by 13
  10. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  11. 2023The Interpolating Information Criterion for Overparameterized Modelscited by 11
  12. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
21 more
  1. 2026RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimizationcited by 11
  2. 2022Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processescited by 10
  3. 2023The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasetscited by 9
  4. 2025Why Less is More (Sometimes): A Theory of Data Curationcited by 8
  5. 2024Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equationscited by 7
  6. 2024Theoretical characterisation of the Gauss-Newton conditioning in Neural Networkscited by 5
  7. 2023A PAC-Bayesian Perspective on the Interpolating Information Criterioncited by 4
  8. 2022Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rulescited by 3
  9. 2023The Universal Statistical Structure and Scaling Laws of Chaos and Turbulencecited by 2
  10. 2025Eigenvalue distribution of the Neural Tangent Kernel in the quadratic scalingcited by 2
  11. 2025Local properties of neural networks through the lens of layer-wise Hessianscited by 2
  12. 2024Implicit Regularization Paths of Weighted Neural Representationscited by 1
  13. 2024Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapescited by 1
  14. 2024Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysiscited by 1
  15. 2025What Scalable Second-Order Information Knows for Pruning at Initializationcited by 1
  16. 2025Fundamental Bias in Inverting Random Sampling Matrices with Application to Sub-sampled Newtoncited by 1
  17. 2025A Teacher-Student Perspective on the Dynamics of Learning Near the Optimal Pointcited by 1
  18. 2024Statistical Mechanics and Artificial Neural Networks: Principles, Models, and Applicationscited by 0
  19. 2025Matricial Free Energy as a Gaussianizing Regularizer: Enhancing Autoencoders for Gaussian Code Generationcited by 0
  20. 2026Fast Gauss-Newton for Multiclass Cross-Entropycited by 0
  21. 2026Eigenanalysis framework for autoregressive neural emulators of multi-scale chaotic dynamicscited by 0