# Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent

arXiv: https://arxiv.org/abs/2606.02596 · HTML: https://mlsanity.com/paper/2606.02596/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 14 references that are in the map.
- First posted on arXiv 22 May 2026.

## Builds on (most cited first, 8 of 8)

- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Shampoo: Preconditioned Stochastic Tensor Optimization](https://mlsanity.com/paper/1802.09568/) (2018)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Old Optimizer, New Norm: An Anthology](https://mlsanity.com/paper/2409.20325/) (2024)
- [A Kronecker-factored approximate Fisher matrix for convolution layers](https://mlsanity.com/paper/1602.01407/) (2016)
- [Products of Many Large Random Matrices and Gradients in Deep Neural Networks](https://mlsanity.com/paper/1812.05994/) (2018)
- [Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks](https://mlsanity.com/paper/2010.04261/) (2020)
- [On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning](https://mlsanity.com/paper/2502.01763/) (2025)

## Abstract (arXiv)

The curvature exponent α in hk propto σkα -- governing how Hessian eigenvalues scale with gradient singular values -- varies systematically across layer types (α ≈ 2 for convolutions, ≈ 1 for transformer attention, < 1 for MLP up-projections). Why? We prove the Spectral Alignment Decomposition: α = 2 + dlogPhik / dlogσk, where Phik measures alignment between Kronecker factor eigenbases and gradient singular directions. This reduces "why does α vary?" to a geometric question we answer for LayerNorm, residual connections, and softmax heads. The decomposition implies a spectral transfer identity s = αγ linking curvature exponent, effective gradient rank-decay γ, and Hessian decay exponent s. The identity is algebraic; its empirical content is that α and γ, fit on independent data (HVPs vs. SVD), recover s to ~2% median error across 93 layers, five architectures, and three datasets -- with no free parameters. A zeta-function bound on participation ratio shows curvature concentrates onto effectively one direction per layer. As a proof of concept, we derive the architecture-adaptive preconditioner T(σ;α) and show that Spectral Newton -- implementing T in the gradient singular basis -- outperforms AdamW on vision benchmarks where α ≈ 2.