mlsanityData to 7 Oct 2026

Analytic Insights into Structure and Rank of Neural Network Hessian Maps

arXiv 2106.16225 ↗First posted 30 Jun 2021cs.LG cs.NE math.STcited by 52

Not on the map. Why?

Abstract (arXiv)

The Hessian of a neural network captures parameter interactions through second-order derivatives of the loss. It is a fundamental object of study, closely tied to various problems in deep learning, including model design, optimization, and generalization. Most prior work has been empirical, typically focusing on low-rank approximations and heuristics that are blind to the network structure. In contrast, we develop theoretical tools to analyze the range of the Hessian map, providing us with a precise understanding of its rank deficiency as well as the structural reasons behind it. This yields exact formulas and tight upper bounds for the Hessian rank of deep linear networks, allowing for an elegant interpretation in terms of rank deficiency. Moreover, we demonstrate that our bounds remain faithful as an estimate of the numerical Hessian rank, for a larger class of models such as rectified and hyperbolic tangent networks. Further, we also investigate the implications of model architecture (e.g.~width, depth, bias) on the rank deficiency. Overall, our work provides novel insights into the source and extent of redundancy in overparameterized networks.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 32 of its 67 references that are in the map. First posted on arXiv 30 Jun 2021.

Builds on

32 of 67 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  9. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  10. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  11. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  12. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
20 more
  1. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  2. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  3. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  4. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  5. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  6. 2016Deep Learning without Poor Local Minimacited by 233
  7. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
  8. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  9. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  10. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  11. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  12. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 89
  13. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  14. 2020Practical Quasi-Newton Methods for Training Deep Neural Networkscited by 67
  15. 2019Shaping the learning landscape in neural networks around wide flat minimacited by 47
  16. 2017Theory of Deep Learning III: explaining the non-overfitting puzzlecited by 32
  17. 2019The asymptotic spectrum of the Hessian of DNN throughout trainingcited by 19
  18. 2021Adaptive Newton Sketch: Linear-time Optimization with Quadratic Convergence and Effective Hessian Dimensionalitycited by 14
  19. 2020Beyond Random Matrix Theory for Deep Networkscited by 11
  20. 2021Lower Bounds on the Generalization Error of Nonlinear Learning Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 5; 2024: 12; 2025: 16; 2026: 13.

52 papers.

Sort:

By area:

  1. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  2. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  3. 2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 42
  4. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  5. 2025HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalizationcited by 15
  6. 2025Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learningcited by 15
  7. 2022On the Overlooked Structure of Stochastic Gradientscited by 13
  8. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  9. 2023Robust low-rank training via approximate orthonormal constraintscited by 12
  10. 2024Neglected Hessian component explains mysteries in Sharpness regularizationcited by 12
  11. 2022Phenomenology of Double Descent in Finite-Width Neural Networkscited by 11
  12. 2026RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimizationcited by 11
38 more
  1. 2023The Hessian perspective into the Nature of Convolutional Neural Networkscited by 10
  2. 2024Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networkscited by 10
  3. 2024Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?cited by 10
  4. 2024What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysiscited by 9
  5. 2025Position: Curvature Matrices Should Be Democratized via Linear Operatorscited by 9
  6. 2024Landscaping Linear Mode Connectivitycited by 7
  7. 2022Federated Optimization of Smooth Loss Functionscited by 6
  8. 2024On Newton's Method to Unlearn Neural Networkscited by 5
  9. 2024Theoretical characterisation of the Gauss-Newton conditioning in Neural Networkscited by 5
  10. 2025Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attackscited by 5
  11. 2025A geometric framework for momentum-based optimizers for low-rank trainingcited by 5
  12. 2023A PAC-Bayesian Perspective on the Interpolating Information Criterioncited by 4
  13. 2025Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobianscited by 4
  14. 2025Optimal Brain Apoptosiscited by 3
  15. 2025TRACE for Tracking the Emergence of Semantic Representations in Transformerscited by 3
  16. 2026Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Lesscited by 3
  17. 2022Gradient Descent for Low-Rank Functionscited by 2
  18. 2024Hallmarks of Optimization Trajectories in Neural Networks: Directional Exploration and Redundancycited by 2
  19. 2025Can Hessian-Based Insights Support Fault Diagnosis in Attention-based Models?cited by 2
  20. 2025Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functionscited by 2
  21. 2026LoRIF: Low-Rank Influence Functions for Scalable Training Data Attributioncited by 2
  22. 2024Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapescited by 1
  23. 2025Accelerating Neural Network Training Along Sharp and Flat Directionscited by 1
  24. 2025When Active Learning Fails, Uncalibrated Out of Distribution Uncertainty Quantification Might Be the Problemcited by 1
  25. 2025A Teacher-Student Perspective on the Dynamics of Learning Near the Optimal Pointcited by 1
  26. 2026Depth, Not Data: An Analysis of Hessian Spectral Bifurcationcited by 1
  27. 2026Wolkowicz-Styan Upper Bound on the Hessian Eigenspectrum for Cross-Entropy Loss in Nonlinear Smooth Neural Networkscited by 1
  28. 2023Towards guarantees for parameter isolation in continual learningcited by 0
  29. 2024Local vs Global continual learningcited by 0
  30. 2024Sampling Foundational Transformer: A Theoretical Perspectivecited by 0
  31. 2025Closing the Curvature Gap: Full Transformer Hessianscited by 0
  32. 2025An Equivariance Toolbox for Learning Dynamicscited by 0
  33. 2026Decentralized SGD with Controlled Disagreement Finds Flatter Minimacited by 0
  34. 2026Curvature-Aligned Probing for Local Loss-Landscape Stabilizationcited by 0
  35. 2026A Bifurcation Theory Framework for Gradient Descent on the Edge of Stabilitycited by 0
  36. 2026On the Oracle Complexity of Interpolation-Based Gradient Descentcited by 0
  37. 2026Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networkscited by 0
  38. 2026Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networkscited by 0

50 most cited of 52 All 52 (JSON)