Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity
Not on the map. Why?
Abstract (arXiv)
Singular learning theory characterises the complexity of a deep network through the geometry of its loss singularities. The local learning coefficient (LLC), the standard estimator of Watanabe's real log canonical threshold (RLCT, λ), reads this geometry as an integrated Bayesian scalar through SGLD, which needs per-task calibration and 104-106 forward-backward passes per checkpoint. We introduce Dead-Direction Signatures (DDS), a family of cheap closed-form spectral readings of singular structure: each reads a network's activation matrix or per-sample-gradient Fisher-Gram at a chosen layer, replacing the SGLD posterior chain with spectral linear algebra. The readings rest on a dead-direction framework that predicts a structural correlation between activation- and Fisher-side spectra at any singular minimum, and a rank-multiplicative volume identity that single-eigenvalue monitors cannot produce: the active-volume logdet+(G) slope counts the dead directions, tracking the rank-deficit r across r ∈ 1,2,3,4 (slope ratios 2.0, 3.1, 4.0 at r=2,3,4 against the predicted 2,3,4), where the smallest eigenvalue is rank-blind. On reduced-rank regression with closed-form λ, calibrated LLC recovers λ at 99% mean and the DDS observables rank-track it at the framework-predicted sign; on a non-linear modular-addition transformer DDS separates dmodel across eighteen orders of magnitude where calibrated LLC at the protocol budget is rank-flat. Complementary to LLC's integrated posterior reading, DDS gives a directional, layer-local handle on a network's dead directions, read in closed form from its activation and gradient spectra.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 19 Jun 2026.
Builds on
25 of 34 references.
References · 25
- 2022Toy Models of Superpositioncited by 971
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
- 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
- 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
- 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
- 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
- 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
- 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
- 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
- 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 146
- 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
13 more
- 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
- 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 91
- 2023Transformers learn through gradual rank increasecited by 52
- 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
- 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
- 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 40
- 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
- 2019Pathological spectra of the Fisher information metric and its variants in deep neural networkscited by 32
- 2024Small Singular Values Matter: A Random Matrix Analysis of Transformer Modelscited by 21
- 2025EigenTrack: Spectral Activation Feature Tracking for Hallucination and Out-of-Distribution Detection in LLMs and VLMscited by 10
- 2026Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scalescited by 6
- 2026NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networkscited by 3
- 2026Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scalecited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.