Distributional Spectral Diagnostics for Localizing Grokking Transitions
Not on the map. Why?
Abstract (arXiv)
In grokking, a model first fits the training data while test accuracy remains low, and only later begins to generalize. We ask whether this transition can be localized from observed training trajectories before the test accuracy rises, and formulate grokking transition localization as a diagnostic problem with an explicit threshold/FPR/lead-time trade-off. Task-dependent observables are summarized as empirical distributions, mapped to Wasserstein/quantile coordinates, and analyzed by Hankel dynamic mode decomposition (DMD); the resulting reconstruction residual, together with spectrum and effective rank, forms the diagnostic output. On held-out modular-addition Transformer runs, the residual achieves AUROC (≈ ) 0.93 for grokking-vs-non-grokking discrimination at the run level; under a fixed sustained-threshold operating rule, true-positive alarms can precede onset, with lead time reported jointly with false-alarm rate and uncertainty intervals. Perturbation experiments show that, in the tested (wd=1) pool, high-residual windows exhibit about (3×) larger short-horizon perturbation deviation than low-residual windows. In a same-data norm-window control, perturbation sensitivity aligns with the residual ordering rather than total-parameter-norm ordering, suggesting that the residual is not merely a total-norm proxy at the window level in the studied (wd=1) dynamics. Norm signals remain strong run-level regime indicators, and log-probability performs best among the observables tested under the current protocol. We position the residual as a window-level monitoring and localization signal in the studied modular-arithmetic Transformer settings, not a universal early-warning predictor or an intervention rule.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 7 May 2026.
Builds on
33 of 46 references.
References · 33
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
- 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
- 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
- 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
21 more
- 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
- 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
- 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
- 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
- 2018Gradient Descent Happens in a Tiny Subspacecited by 208
- 2020Feature Learning in Infinite-Width Neural Networkscited by 182
- 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
- 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
- 2023Explaining grokking through circuit efficiencycited by 105
- 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
- 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
- 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
- 2025Grokking at the Edge of Numerical Stabilitycited by 41
- 2020Optimizing Neural Networks via Koopman Operator Theorycited by 32
- 2023The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifoldcited by 20
- 2025The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifoldcited by 8
- 2020Accelerating Training in Artificial Neural Networks with Dynamic Mode Decompositioncited by 7
- 2022QuACK: Accelerating Gradient-Based Quantum Optimization with Koopman Operator Learningcited by 7
- 2023Identifying Equivalent Training Dynamicscited by 7
- 2021Low Dimensional Landscape Hypothesis is True: DNNs can be Trained in Tiny Subspacescited by 5
- 2023Enhancing Neural Training via a Correlated Dynamics Modelcited by 5
Built on by
Citing papers per year: 2026: 2.
2 papers.