mlsanityData to 7 Oct 2026

Unveiling the Hessian's Connection to the Decision Boundary

arXiv 2306.07104 ↗First posted 12 Jun 2023cs.LG cond-mat.dis-nn stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Understanding the properties of well-generalizing minima is at the heart of deep learning research. On the one hand, the generalization of neural networks has been connected to the decision boundary complexity, which is hard to study in the high-dimensional input space. Conversely, the flatness of a minimum has become a controversial proxy for generalization. In this work, we provide the missing link between the two approaches and show that the Hessian top eigenvectors characterize the decision boundary learned by the neural network. Notably, the number of outliers in the Hessian spectrum is proportional to the complexity of the decision boundary. Based on this finding, we provide a new and straightforward approach to studying the complexity of a high-dimensional decision boundary; show that this connection naturally inspires a new generalization measure; and finally, we develop a novel margin estimation technique which, in combination with the generalization measure, precisely identifies minima with simple wide-margin boundaries. Overall, this analysis establishes the connection between the Hessian and the decision boundary and provides a new method to identify minima with simple wide-margin decision boundaries.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 34 of its 52 references that are in the map. First posted on arXiv 12 Jun 2023.

Builds on

34 of 52 references.

References · 34

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  3. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  6. 2017A Closer Look at Memorization in Deep Networkscited by 936
  7. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  8. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  9. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  10. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  11. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  12. 2014The Loss Surfaces of Multilayer Networkscited by 278
22 more
  1. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  2. 2019Explanations can be manipulated and geometry is to blamecited by 215
  3. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  4. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  5. 2018Robustness via curvature regularization, and vice versacited by 174
  6. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  7. 2019Adversarial Robustness through Local Linearizationcited by 146
  8. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  9. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  10. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  11. 2020Relative Flatness and Generalizationcited by 88
  12. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
  13. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  14. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  15. 2019Bad Global Minima Exist and SGD Can Reach Themcited by 49
  16. 2019Emergent properties of the local geometry of neural loss landscapescited by 46
  17. 2020Towards Robust Explanations for Deep Neural Networkscited by 43
  18. 2021Hessian-Aware Pruning and Optimal Neural Implantcited by 43
  19. 2022On the Generalization Mystery in Deep Learningcited by 25
  20. 2019Negative eigenvalues of the Hessian in deep neural networkscited by 20
  21. 2018Topological Data Analysis of Decision Boundaries with Application to Model Selectioncited by 19
  22. 2020Analysis of Generalizability of Deep Neural Networks Based on the Complexity of Decision Boundarycited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 1.

5 papers.

  1. 2024Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularizationcited by 6
  2. 2024Deconstructing the Goldilocks Zone of Neural Network Initializationcited by 4
  3. 2024Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Featurescited by 1
  4. 2025Exploring Cross-Stage Adversarial Transferability in Class-Incremental Continual Learningcited by 0
  5. 2026Softly Induced Functional Simplicity: Implications for Neural Network Generalisation, Robustness, and Distillationcited by 0