# Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Training

arXiv: https://arxiv.org/abs/2603.28964 · HTML: https://mlsanity.com/paper/2603.28964/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 38 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Builds on (most cited first, 12 of 12)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017)
- [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018)
- [Tensor Programs II: Neural Tangent Kernel for Any Architecture](https://mlsanity.com/paper/2006.14548/) (2020)
- [Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking](https://mlsanity.com/paper/2602.16746/) (2026)
- [The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structure](https://mlsanity.com/paper/2602.18523/) (2026)
- [Early-Warning Signals of Grokking via Loss-Landscape Geometry](https://mlsanity.com/paper/2602.16967/) (2026)
- [Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales](https://mlsanity.com/paper/2603.15678/) (2026)
- [Optimizer-Induced Low-Dimensional Drift and Transverse Dynamics in Transformer Training](https://mlsanity.com/paper/2602.23696/) (2026)
- [Global Low-Rank, Local Full-Rank: The Holographic Encoding of Learned Algorithms](https://mlsanity.com/paper/2602.18649/) (2026)

## Built on by (most cited first, 4 of 4)

- [Spectral Edge Dynamics Reveal Functional Modes of Learning](https://mlsanity.com/paper/2604.06256/) (2026)
- [Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics](https://mlsanity.com/paper/2605.20441/) (2026)
- [The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression](https://mlsanity.com/paper/2604.07380/) (2026)
- [Gradient-Direction Sensitivity Reveals Linear-Centroid Coupling Hidden by Optimizer Trajectories](https://mlsanity.com/paper/2604.25143/) (2026)

## Abstract (arXiv)

We develop the spectral edge analysis: phase transitions in neural network training -- grokking, capability gains, loss plateaus -- are controlled by the spectral gap of the rolling-window Gram matrix of parameter updates. In the extreme aspect ratio regime (parameters P ~ 108, window W ~ 10), the classical BBP detection threshold is vacuous; the operative structure is the intra-signal gap separating dominant from subdominant modes at position k^* = argmax, σj/σj+1. From three assumptions we derive: (i) gap dynamics governed by a Dyson-type ODE with curvature asymmetry, damping, and gradient driving; (ii) a spectral loss decomposition linking each mode's learning contribution to its Davis--Kahan stability coefficient; (iii) the Gap Maximality Principle, showing that k^* is the unique dynamically privileged position -- its collapse is the only one that disrupts learning, and it sustains itself through an α-feedback loop requiring no assumption on the optimizer. The adiabatic parameter A = |Delta G|F / (η, g2) controls circuit stability: A ll 1 (plateau), A ~ 1 (phase transition), A gg 1 (forgetting). Tested across six model families (150K--124M parameters): gap dynamics precede every grokking event (24/24 with weight decay, 1/24 without), the gap position is optimizer-dependent (Muon: k^*=1, AdamW: k^*=2 on the same model), and 19/20 quantitative predictions are confirmed. The framework is consistent with the edge of stability, Tensor Programs, Dyson Brownian motion, the Lottery Ticket Hypothesis, and neural scaling laws.