# Distributional Spectral Diagnostics for Localizing Grokking Transitions arXiv: https://arxiv.org/abs/2605.08237 · HTML: https://mlsanity.com/paper/2605.08237/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Builds on 33 of its 46 references that are in the map. - First posted on arXiv 7 May 2026. ## Builds on (most cited first, 30 of 33) - [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018) - [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023) - [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019) - [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018) - [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020) - [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022) - [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018) - [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021) - [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018) - [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018) - [Kernel and Rich Regimes in Overparametrized Models](https://mlsanity.com/paper/2002.09277/) (2020) - [Stochastic Gradient Descent as Approximate Bayesian Inference](https://mlsanity.com/paper/1704.04289/) (2017) - [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019) - [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017) - [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022) - [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018) - [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018) - [Feature Learning in Infinite-Width Neural Networks](https://mlsanity.com/paper/2011.14522/) (2020) - [Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability](https://mlsanity.com/paper/2209.15594/) (2022) - [Explaining grokking through circuit efficiency](https://mlsanity.com/paper/2309.02390/) (2023) - [Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach](https://mlsanity.com/paper/1805.00915/) (2018) - [The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon](https://mlsanity.com/paper/2206.04817/) (2022) - [Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking](https://mlsanity.com/paper/2311.18817/) (2023) - [Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians](https://mlsanity.com/paper/1901.08244/) (2019) - [Grokking at the Edge of Numerical Stability](https://mlsanity.com/paper/2501.04697/) (2025) - [Optimizing Neural Networks via Koopman Operator Theory](https://mlsanity.com/paper/2006.02361/) (2020) - [The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold](https://mlsanity.com/paper/2305.01604/) (2023) - [Identifying Equivalent Training Dynamics](https://mlsanity.com/paper/2302.09160/) (2023) - [The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold](https://mlsanity.com/paper/2511.01938/) (2025) - [Accelerating Training in Artificial Neural Networks with Dynamic Mode Decomposition](https://mlsanity.com/paper/2006.14371/) (2020) ## Built on by (most cited first, 2 of 2) - [Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics](https://mlsanity.com/paper/2605.20441/) (2026) - [At-Grok Is Not Converged:A Measurement-Validity Audit for Grokking Representation Metrics](https://mlsanity.com/paper/2607.06639/) (2026) ## Abstract (arXiv) In grokking, a model first fits the training data while test accuracy remains low, and only later begins to generalize. We ask whether this transition can be localized from observed training trajectories before the test accuracy rises, and formulate grokking transition localization as a diagnostic problem with an explicit threshold/FPR/lead-time trade-off. Task-dependent observables are summarized as empirical distributions, mapped to Wasserstein/quantile coordinates, and analyzed by Hankel dynamic mode decomposition (DMD); the resulting reconstruction residual, together with spectrum and effective rank, forms the diagnostic output. On held-out modular-addition Transformer runs, the residual achieves AUROC (≈ ) 0.93 for grokking-vs-non-grokking discrimination at the run level; under a fixed sustained-threshold operating rule, true-positive alarms can precede onset, with lead time reported jointly with false-alarm rate and uncertainty intervals. Perturbation experiments show that, in the tested (wd=1) pool, high-residual windows exhibit about (3×) larger short-horizon perturbation deviation than low-residual windows. In a same-data norm-window control, perturbation sensitivity aligns with the residual ordering rather than total-parameter-norm ordering, suggesting that the residual is not merely a total-norm proxy at the window level in the studied (wd=1) dynamics. Norm signals remain strong run-level regime indicators, and log-probability performs best among the observables tested under the current protocol. We position the residual as a window-level monitoring and localization signal in the studied modular-arithmetic Transformer settings, not a universal early-warning predictor or an intervention rule.