# Dead Directions: Geometric Singular Learning

arXiv: https://arxiv.org/abs/2606.05957 · HTML: https://mlsanity.com/paper/2606.05957/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 54 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Builds on (most cited first, 30 of 34)

- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://mlsanity.com/paper/2202.08906/) (2022)
- [Massive Activations in Large Language Models](https://mlsanity.com/paper/2402.17762/) (2024)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [Mesh-TensorFlow: Deep Learning for Supercomputers](https://mlsanity.com/paper/1811.02084/) (2018)
- [Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse](https://mlsanity.com/paper/2206.03126/) (2022)
- [Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity](https://mlsanity.com/paper/2106.09524/) (2021)
- [Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics](https://mlsanity.com/paper/2012.04728/) (2020)
- [The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon](https://mlsanity.com/paper/2206.04817/) (2022)
- [Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks](https://mlsanity.com/paper/2105.02716/) (2021)
- [Grokking at the Edge of Numerical Stability](https://mlsanity.com/paper/2501.04697/) (2025)
- [Loss Landscape Degeneracy and Stagewise Development in Transformers](https://mlsanity.com/paper/2402.02364/) (2024)
- [Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient](https://mlsanity.com/paper/2410.02984/) (2024)
- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [Peri-LN: Revisiting Normalization Layer in the Transformer Architecture](https://mlsanity.com/paper/2502.02732/) (2025)
- [Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition](https://mlsanity.com/paper/2310.06301/) (2023)
- [Bayesian Influence Functions for Hessian-Free Data Attribution](https://mlsanity.com/paper/2509.26544/) (2025)
- [Using Degeneracy in the Loss Landscape for Mechanistic Interpretability](https://mlsanity.com/paper/2405.10927/) (2024)
- [Structural Inference: Interpreting Small Language Models with Susceptibilities](https://mlsanity.com/paper/2504.18274/) (2025)
- [The Z-loss: a shift and scale invariant classification loss belonging to the Spherical Family](https://mlsanity.com/paper/1604.08859/) (2016)
- [Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers](https://mlsanity.com/paper/2605.18106/) (2026)
- [The Loss Kernel: A Geometric Probe for Deep Learning Interpretability](https://mlsanity.com/paper/2509.26537/) (2025)
- [Influence Dynamics and Stagewise Data Attribution](https://mlsanity.com/paper/2510.12071/) (2025)
- [Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networks](https://mlsanity.com/paper/2305.05089/) (2023)
- [Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory](https://mlsanity.com/paper/2510.12077/) (2025)
- [Programs as Singularities](https://mlsanity.com/paper/2504.08075/) (2025)
- [Towards Spectroscopy: Susceptibility Clusters in Language Models](https://mlsanity.com/paper/2601.12703/) (2026)

## Built on by (most cited first, 1 of 1)

- [Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks](https://mlsanity.com/paper/2606.29176/) (2026)

## Abstract (arXiv)

Singular learning theory and information geometry study the same spaces: the former in resolved coordinates, the latter in original coordinates under a non-degeneracy assumption that overparameterised models violate. This paper carries one direction of the bridge between them, from Watanabe's invariants to Fisher geometry, through one primitive, the dead direction: a unit vector along which the Fisher metric degenerates, equivalently a direction crossing the analytic singular set along which the KL divergence keeps a zero of high order, its KL order set by how fast that divergence vanishes. Our central result recovers the KL order as the decay rate of the directional Fisher quadratic form approaching the singularity, in original coordinates, without a Hironaka resolution. A selection rule on smooth fibres translates this rate into Watanabe's single-direction contribution to the real log canonical threshold, and the recovery extends to multi-component crossings, multiplicity m, the singular fluctuation nu, prior-RLCT shifts, and tempered posteriors. We then carry the rate into a deep network: a multi-layer K-FAC factorisation writes each Fisher block as a product of activation- and gradient-side rates with a duality between them, instantiated at residual streams, layer normalisation, and attention. A quotient theorem carries the rate to the gauge quotient for optimizers whose update commutes with the group action; Adam's per-coordinate preconditioner fails that condition, so we construct DDCAdam, an equivariant Adam-family preconditioner, and prove the quotient rate along its trajectory. The result is a trajectory-rate readout of Watanabe's triple (λ, m, nu) from one checkpoint's forward and backward passes, without posterior sampling.