Depth, Not Data: An Analysis of Hessian Spectral Bifurcation
Not on the map. Why?
Abstract (arXiv)
The eigenvalue distribution of the Hessian matrix plays a crucial role in understanding the optimization landscape of deep neural networks. Prior work has attributed the well-documented ``bulk-and-spike'' spectral structure, where a few dominant eigenvalues are separated from a bulk of smaller ones, to the imbalance in the data covariance matrix. In this work, we challenge this view by demonstrating that such spectral Bifurcation can arise purely from the network architecture, independent of data imbalance. Specifically, we analyze a deep linear network setup and prove that, even when the data covariance is perfectly balanced, the Hessian still exhibits a Bifurcation eigenvalue structure: a dominant cluster and a bulk cluster. Crucially, we establish that the ratio between dominant and bulk eigenvalues scales linearly with the network depth. This reveals that the spectral gap is strongly affected by the network architecture rather than solely by data distribution. Our results suggest that both model architecture and data characteristics should be considered when designing optimization algorithms for deep networks.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 21 of its 25 references that are in the map. First posted on arXiv 31 Jan 2026.
Builds on
21 of 25 references.
References · 21
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
- 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
- 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
- 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
- 2017How to Escape Saddle Points Efficientlycited by 284
- 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
- 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
- 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
- 2018Gradient Descent Happens in a Tiny Subspacecited by 208
- 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
9 more
- 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
- 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
- 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
- 2017A Random Matrix Approach to Neural Networkscited by 80
- 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
- 2024Does SGD really happen in tiny subspaces?cited by 35
- 2025Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfercited by 17
- 2025Models of Heavy-Tailed Mechanistic Universalitycited by 9
- 2024Theoretical characterisation of the Gauss-Newton conditioning in Neural Networkscited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.