mlsanityData to 7 Oct 2026

Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent

arXiv 2606.02596 ↗First posted 22 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The curvature exponent α in hk propto σkα -- governing how Hessian eigenvalues scale with gradient singular values -- varies systematically across layer types (α ≈ 2 for convolutions, ≈ 1 for transformer attention, < 1 for MLP up-projections). Why? We prove the Spectral Alignment Decomposition: α = 2 + dlogPhik / dlogσk, where Phik measures alignment between Kronecker factor eigenbases and gradient singular directions. This reduces "why does α vary?" to a geometric question we answer for LayerNorm, residual connections, and softmax heads. The decomposition implies a spectral transfer identity s = αγ linking curvature exponent, effective gradient rank-decay γ, and Hessian decay exponent s. The identity is algebraic; its empirical content is that α and γ, fit on independent data (HVPs vs. SVD), recover s to ~2% median error across 93 layers, five architectures, and three datasets -- with no free parameters. A zeta-function bound on participation ratio shows curvature concentrates onto effectively one direction per layer. As a proof of concept, we derive the architecture-adaptive preconditioner T(σ;α) and show that Spectral Newton -- implementing T in the gradient singular basis -- outperforms AdamW on vision benchmarks where α ≈ 2.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 8 of its 14 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

8 of 14 references.

References · 8

  1. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  2. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  3. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  4. 2024Old Optimizer, New Norm: An Anthologycited by 173
  5. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  6. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  7. 2020Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networkscited by 45
  8. 2025On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learningcited by 12

Built on by

Not cited yet.