Measuring Dead Directions: Decomposing and Classifying Singular Structure off Canonical Alignment
Not on the map. Why?
Abstract (arXiv)
We give a descent-free, alignment-free measurement of singular structure on trained networks. At a single frozen checkpoint the read recovers the order k of each dead direction from the directional-Fisher rate, the master invariant from which the per-direction learning coefficient 1/(2k) follows exactly, in whatever basis the optimizer left. The same read classifies each direction, separating a genuine singularity, whose order the architecture fixes, from a flat gauge symmetry; the directional-Fisher magnitude settles the cases the order cannot. A pluggable detector supplies the directions for transformer, convolutional, and normalisation layers. The read recovers the architecture-predicted order across constructed cells and trained networks, including a fine-tuned vision transformer whose dead structure is the LayerNorm-kernel gauge and a from-scratch one whose compressed MLP forms a node-death at its activation order. Where the singular structure enumerates, the per-direction orders assemble, through the typed intersection of the loci, into the global coefficient (λ, m) matching the closed form. The method removes the canonical-alignment and descent preconditions of the underlying rate result, turning order-recovery into a deterministic, architecture-general reading. We then map its reach into the Watanabe triple: the order determines the universal singular fluctuation nu(k), though a trained network's realized nu falls below it as the live structure absorbs the dead direction's data fluctuation, and the multiplicity recovers from the dominant structure under a single-locus assumption.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 31 references that are in the map. First posted on arXiv 1 Jul 2026.
Builds on
14 of 31 references.
References · 14
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
- 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
- 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
- 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
- 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 40
- 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
- 2024Using Degeneracy in the Loss Landscape for Mechanistic Interpretabilitycited by 13
- 2023Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networkscited by 6
- 2026Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scalecited by 3
Built on by
Not cited yet.