At-Grok Is Not Converged:A Measurement-Validity Audit for Grokking Representation Metrics
Not on the map. Why?
Abstract (arXiv)
On modular arithmetic, a network's embedding keeps compressing for tens of thousands of steps after it has already generalized. Reading effective rank at the grokking transition overstates the converged value by 3-5x on an MLP, and by 1.3-1.5x on a transformer trained to convergence; on the MLP it also erases which cells compress at all. Compression lags the accuracy transition by an amount on the order of the time-to-grok, at least 10,000 steps, rather than coinciding with it. A one-variable ablation shows what sets the lag size: adding LayerNorm to an otherwise identical transformer moves the fraction of compression done by the grok step from 0.87 to 0.25, and a pre-registered control rules out scale invariance as the mechanism. We package this as an audit that separates onset from compression, flags censoring, excludes boundary cells that never fully generalize, and checks that the reference floor has plateaued, with an adversarial suite that caught a false-confidence bug in our own branch. A secondary, MLP-specific depth law linking norm budget to converged floor fails a generality test on a transformer and flips sign under free weight decay. Code and the toolkit are released.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 7 Jul 2026.
Builds on
21 of 33 references.
References · 21
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
- 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
- 2023Explaining grokking through circuit efficiencycited by 105
- 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
- 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
- 2022Relating Regularization and Generalization through the Intrinsic Dimension of Activationscited by 9
- 2026The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structurecited by 8
- 2025Mechanistic Insights into Grokking from the Embedding Layercited by 6
- 2026Early-Warning Signals of Grokking via Loss-Landscape Geometrycited by 6
- 2026The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalizationcited by 6
- 2026A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularizationcited by 5
9 more
- 2026Noise-Driven Escape from Metastable Phases explains Grokking in Deep Neural Networkscited by 5
- 2026Grokking as Dimensional Phase Transition in Neural Networkscited by 4
- 2026Late-Stage Generalization Collapse in Grokking: Detecting anti-grokking with Weightwatchercited by 2
- 2026ILDR: Geometric Early Detection of Grokkingcited by 2
- 2026Topological Signatures of Grokkingcited by 2
- 2026Distributional Spectral Diagnostics for Localizing Grokking Transitionscited by 2
- 2026Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnosticscited by 2
- 2026Circuit Synchronization Precedes Generalization: A Causal Precursor to Grokkingcited by 2
- 2026Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric Viewcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.