# The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression

arXiv: https://arxiv.org/abs/2604.07380 · HTML: https://mlsanity.com/paper/2604.07380/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Builds on (most cited first, 14 of 14)

- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [Gradient Descent Maximizes the Margin of Homogeneous Neural Networks](https://mlsanity.com/paper/1906.05890/) (2019)
- [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018)
- [Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking](https://mlsanity.com/paper/2311.18817/) (2023)
- [Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking](https://mlsanity.com/paper/2602.16746/) (2026)
- [The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structure](https://mlsanity.com/paper/2602.18523/) (2026)
- [Low-Dimensional Execution Manifolds in Transformer Learning Dynamics: Evidence from Modular Arithmetic Tasks](https://mlsanity.com/paper/2602.10496/) (2026)
- [Early-Warning Signals of Grokking via Loss-Landscape Geometry](https://mlsanity.com/paper/2602.16967/) (2026)
- [Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Training](https://mlsanity.com/paper/2603.28964/) (2026)
- [Spectral Edge Dynamics Reveal Functional Modes of Learning](https://mlsanity.com/paper/2604.06256/) (2026)

## Built on by (most cited first, 1 of 1)

- [Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics](https://mlsanity.com/paper/2605.20441/) (2026)

## Abstract (arXiv)

We decompose the spectral edge -- the dominant direction of the Gram matrix of parameter updates -- into its gradient and weight-decay components during grokking in two sequence tasks (Dyck-1 and SCAN). We find a sharp two-phase lifecycle: before grokking the edge is gradient-driven and functionally active; at grokking, gradient and weight decay align, and the edge becomes a compression axis that is perturbation-flat yet ablation-critical (>4000x more impactful than random directions). Three universality classes emerge (functional, mixed, compression), predicted by the gap flow equation. Nonlinear probes show information is re-encoded, not lost (MLP R2=0.99 where linear R2=0.86), and removing weight decay post-grok reverses compression while preserving the algorithm.