mlsanityData to 14 Oct 2026

Dead Directions: Geometric Singular Learning

arXiv 2606.05957 ↗First posted 4 Jun 2026cs.LG stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

Singular learning theory and information geometry study the same spaces: the former in resolved coordinates, the latter in original coordinates under a non-degeneracy assumption that overparameterised models violate. This paper carries one direction of the bridge between them, from Watanabe's invariants to Fisher geometry, through one primitive, the dead direction: a unit vector along which the Fisher metric degenerates, equivalently a direction crossing the analytic singular set along which the KL divergence keeps a zero of high order, its KL order set by how fast that divergence vanishes. Our central result recovers the KL order as the decay rate of the directional Fisher quadratic form approaching the singularity, in original coordinates, without a Hironaka resolution. A selection rule on smooth fibres translates this rate into Watanabe's single-direction contribution to the real log canonical threshold, and the recovery extends to multi-component crossings, multiplicity m, the singular fluctuation nu, prior-RLCT shifts, and tempered posteriors. We then carry the rate into a deep network: a multi-layer K-FAC factorisation writes each Fisher block as a product of activation- and gradient-side rates with a duality between them, instantiated at residual streams, layer normalisation, and attention. A quotient theorem carries the rate to the gauge quotient for optimizers whose update commutes with the group action; Adam's per-coordinate preconditioner fails that condition, so we construct DDCAdam, an equivariant Adam-family preconditioner, and prove the quotient rate along its trajectory. The result is a trajectory-rate readout of Watanabe's triple (λ, m, nu) from one checkpoint's forward and backward passes, without posterior sampling.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 34 of its 54 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

34 of 54 references.

References · 34

  1. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  2. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  3. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  4. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  6. 2024Massive Activations in Large Language Modelscited by 294
  7. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
  8. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  9. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  10. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  11. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
  12. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 94
22 more
  1. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  2. 2025Grokking at the Edge of Numerical Stabilitycited by 46
  3. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 40
  4. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
  5. 2025Symmetry in Neural Network Parameter Spacescited by 36
  6. 2025Peri-LN: Revisiting Normalization Layer in the Transformer Architecturecited by 34
  7. 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 29
  8. 2025Bayesian Influence Functions for Hessian-Free Data Attributioncited by 14
  9. 2024Using Degeneracy in the Loss Landscape for Mechanistic Interpretabilitycited by 13
  10. 2025Structural Inference: Interpreting Small Language Models with Susceptibilitiescited by 13
  11. 2016The Z-loss: a shift and scale invariant classification loss belonging to the Spherical Familycited by 12
  12. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 11
  13. 2025The Loss Kernel: A Geometric Probe for Deep Learning Interpretabilitycited by 9
  14. 2025Influence Dynamics and Stagewise Data Attributioncited by 7
  15. 2023Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networkscited by 6
  16. 2025Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theorycited by 5
  17. 2025Programs as Singularitiescited by 4
  18. 2026Towards Spectroscopy: Susceptibility Clusters in Language Modelscited by 4
  19. 2025Modes of Sequence Models and Learning Coefficientscited by 3
  20. 2025How do simple rotations affect the implicit bias of Adam?cited by 3
  21. 2026Scale redundancy and soft gauge fixing in positively homogeneous neural networkscited by 2
  22. 2023Blow-up Algorithm for Sum-of-Products Polynomials and Real Log Canonical Thresholdscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.