# Early-Warning Signals of Grokking via Loss-Landscape Geometry

arXiv: https://arxiv.org/abs/2602.16967 · HTML: https://mlsanity.com/paper/2602.16967/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 24 references that are in the map.
- First posted on arXiv 19 Feb 2026.

## Builds on (most cited first, 9 of 9)

- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Are Transformers universal approximators of sequence-to-sequence functions?](https://mlsanity.com/paper/1912.10077/) (2019)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit](https://mlsanity.com/paper/2207.08799/) (2022)
- [Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking](https://mlsanity.com/paper/2311.18817/) (2023)
- [Grokking of Hierarchical Structure in Vanilla Transformers](https://mlsanity.com/paper/2305.18741/) (2023)
- [Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking](https://mlsanity.com/paper/2602.16746/) (2026)
- [Low-Dimensional Execution Manifolds in Transformer Learning Dynamics: Evidence from Modular Arithmetic Tasks](https://mlsanity.com/paper/2602.10496/) (2026)

## Built on by (most cited first, 7 of 7)

- [Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales](https://mlsanity.com/paper/2603.15678/) (2026)
- [Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Training](https://mlsanity.com/paper/2603.28964/) (2026)
- [Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics](https://mlsanity.com/paper/2605.20441/) (2026)
- [Global Low-Rank, Local Full-Rank: The Holographic Encoding of Learned Algorithms](https://mlsanity.com/paper/2602.18649/) (2026)
- [The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression](https://mlsanity.com/paper/2604.07380/) (2026)
- [At-Grok Is Not Converged:A Measurement-Validity Audit for Grokking Representation Metrics](https://mlsanity.com/paper/2607.06639/) (2026)
- [An Exploratory Replica-Overlap Probe of the Grokking Transition](https://mlsanity.com/paper/2609.25634/) (2026)

## Abstract (arXiv)

Grokking -- the abrupt transition from memorization to generalization after prolonged training -- has been linked to confinement on low-dimensional execution manifolds in modular arithmetic. Whether this mechanism extends beyond arithmetic remains open. We study two sequence-learning benchmarks: SCAN compositional generalization and Dyck-1 depth prediction. Across both tasks and a wide range of learning rates, the commutator defect -- a curvature measure derived from non-commuting gradient updates -- rises well before generalization, with lead times following a superlinear power law (alpha approximately 1.18 for SCAN, approximately 1.13 for Dyck), consistent with prior results on modular arithmetic. Weight-space PCA reveals that spectral concentration is not a universal precursor; the commutator defect is. Causal interventions demonstrate a mechanistic role: amplifying non-commutativity accelerates grokking (roughly 32% on SCAN, roughly 50% on Dyck), while suppressing orthogonal gradient flow delays or prevents it. The three task families form a spectrum of causal sensitivity -- modular arithmetic is rigid, Dyck is responsive, SCAN is intermediate -- yet suppression delays or prevents grokking in all cases, establishing necessity as a universal finding. These results identify the commutator defect as a robust, architecture-agnostic, causally implicated early-warning signal for delayed generalization in transformers.