# Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales

arXiv: https://arxiv.org/abs/2603.15678 · HTML: https://mlsanity.com/paper/2603.15678/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 14 Mar 2026.

## Builds on (most cited first, 16 of 16)

- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019)
- [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017)
- [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworks](https://mlsanity.com/paper/2303.11873/) (2023)
- [Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking](https://mlsanity.com/paper/2311.18817/) (2023)
- [Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking](https://mlsanity.com/paper/2602.16746/) (2026)
- [The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structure](https://mlsanity.com/paper/2602.18523/) (2026)
- [Low-Dimensional Execution Manifolds in Transformer Learning Dynamics: Evidence from Modular Arithmetic Tasks](https://mlsanity.com/paper/2602.10496/) (2026)
- [Early-Warning Signals of Grokking via Loss-Landscape Geometry](https://mlsanity.com/paper/2602.16967/) (2026)
- [Optimizer-Induced Low-Dimensional Drift and Transverse Dynamics in Transformer Training](https://mlsanity.com/paper/2602.23696/) (2026)

## Built on by (most cited first, 6 of 6)

- [Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Training](https://mlsanity.com/paper/2603.28964/) (2026)
- [Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers](https://mlsanity.com/paper/2605.24059/) (2026)
- [Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale](https://mlsanity.com/paper/2606.19491/) (2026)
- [Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity](https://mlsanity.com/paper/2606.21158/) (2026)
- [When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures](https://mlsanity.com/paper/2606.02378/) (2026)
- [The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry](https://mlsanity.com/paper/2604.22778/) (2026)

## Abstract (arXiv)

Despite hundreds of millions of parameters, transformer training trajectories evolve within only a few coherent directions. We introduce Spectral Edge Dynamics (SED) to quantify this structure: a rolling-window SVD of parameter updates reveals a sharp boundary -- the spectral edge -- between coherent optimization directions and stochastic noise, identified via the maximum consecutive singular value ratio σk / σk+1. Across a 51M-parameter TinyStories model (4 seeds) and GPT-2 124M under distribution shift, the spectral edge exhibits a universal three-phase pattern (rise, plateau, collapse). The effective signal rank adapts to task complexity (k^* = 2 at 51M, k^* = 3 at 124M), and the directional coupling between spectral geometry and validation loss reverses with window size -- a lag flip reflecting the timescale of trajectory integration. Johnson--Lindenstrauss projection to d = 10W dimensions (e.g., d = 100 for W = 10) preserves the spectral gap within 5.7%, making the framework applicable to models of arbitrary scale. In companion work, the same spectral geometry provides early-warning signals of grokking -- predicting generalization 600--1,700 steps before it occurs across modular arithmetic, Dyck languages, and the SCAN benchmark.