Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent
Not on the map. Why?
Abstract (arXiv)
The curvature exponent α in hk propto σkα -- governing how Hessian eigenvalues scale with gradient singular values -- varies systematically across layer types (α ≈ 2 for convolutions, ≈ 1 for transformer attention, < 1 for MLP up-projections). Why? We prove the Spectral Alignment Decomposition: α = 2 + dlogPhik / dlogσk, where Phik measures alignment between Kronecker factor eigenbases and gradient singular directions. This reduces "why does α vary?" to a geometric question we answer for LayerNorm, residual connections, and softmax heads. The decomposition implies a spectral transfer identity s = αγ linking curvature exponent, effective gradient rank-decay γ, and Hessian decay exponent s. The identity is algebraic; its empirical content is that α and γ, fit on independent data (HVPs vs. SVD), recover s to ~2% median error across 93 layers, five architectures, and three datasets -- with no free parameters. A zeta-function bound on participation ratio shows curvature concentrates onto effectively one direction per layer. As a proof of concept, we derive the architecture-adaptive preconditioner T(σ;α) and show that Spectral Newton -- implementing T in the gradient singular basis -- outperforms AdamW on vision benchmarks where α ≈ 2.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 8 of its 14 references that are in the map. First posted on arXiv 22 May 2026.
Builds on
8 of 14 references.
References · 8
- 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
- 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
- 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
- 2024Old Optimizer, New Norm: An Anthologycited by 173
- 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
- 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
- 2020Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networkscited by 45
- 2025On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learningcited by 12
Built on by
Not cited yet.