# Structure-Specific Representational Priors Causally Control the Grokking Delay

arXiv: https://arxiv.org/abs/2607.04333 · HTML: https://mlsanity.com/paper/2607.04333/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 17 references that are in the map.
- First posted on arXiv 5 Jul 2026.

## Builds on (most cited first, 17 of 17)

- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Omnigrok: Grokking Beyond Algorithmic Data](https://mlsanity.com/paper/2210.01117/) (2022)
- [Explaining grokking through circuit efficiency](https://mlsanity.com/paper/2309.02390/) (2023)
- [Grokking modular arithmetic](https://mlsanity.com/paper/2301.02679/) (2023)
- [Grokfast: Accelerated Grokking by Amplifying Slow Gradients](https://mlsanity.com/paper/2405.20233/) (2024)
- [Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds](https://mlsanity.com/paper/2605.09724/) (2026)
- [The Weight Norm Sets the Grokking Timescale: A Causal Delay Law](https://mlsanity.com/paper/2606.13753/) (2026)
- [NeuralGrok: Accelerate Grokking by Neural Gradient Transformation](https://mlsanity.com/paper/2504.17243/) (2025)
- [Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent](https://mlsanity.com/paper/2605.27078/) (2026)
- [Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmetic](https://mlsanity.com/paper/2606.08985/) (2026)
- [Learned Structure Shows Up Earlier in the Task's Own Basis: A Controlled Look at a Synchronisation Precursor in Grokking](https://mlsanity.com/paper/2606.12966/) (2026)
- [What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy](https://mlsanity.com/paper/2606.18465/) (2026)
- [Radial Suppression Accelerates Algorithmic Generalization: A Geometric Analysis of Delayed Generalization](https://mlsanity.com/paper/2606.32000/) (2026)

## Built on by (most cited first, 3 of 3)

- [Capability Emergence Can Be Forecast: Per-Seed, In Advance, With Calibrated Intervals, Certified False Alarms, and a Blind Pre-Registered Gate](https://mlsanity.com/paper/2609.19000/) (2026)
- [Small Enough to Know Everything: The Fully-Enumerable Transformer as an Instrument for the Science of Delayed Generalization](https://mlsanity.com/paper/2609.20166/) (2026)
- [Decodable In-Context State and Model Output Across Training](https://mlsanity.com/paper/2609.31401/) (2026)

## Abstract (arXiv)

Grokking -- generalization long after training-set interpolation -- has been accelerated by structure-agnostic interventions (gradient filtering, weight-norm clamping, geometric penalties). Whether the delay specifically measures the time to form task-structured representations has remained observational. We test it causally by injecting representational priors of varying content into a one-layer transformer learning modular addition, via a supervised-contrastive loss whose positives encode (i) the task's true structure ((a+b) bmod p), (ii) a coherent-but-wrong sibling ((a-b) bmod p), or (iii) a random partition -- all with identical loss form, strength, class sizes, and geometry. Whether generalization occurs follows a clean gradation: true 22/30 runs, sibling (same periodic features, wrong combination) 14/15, random (only memorizable) 0/20 (Fisher p=1.3×10-7). A weight-norm-matched control replaying the norm trajectory onto plain cross-entropy generalizes 0/15, ruling out the norm as mediator. Probes show structure formation precedes and predicts generalization in all runs. Only the true structure also accelerates grokking (up to 2.75×), but this is dose-dependent and bimodal. We then confirm the mechanism by prediction: because the acceleration is gated by a weight-norm side-effect, clamping the norm during training yields a reliable, standalone accelerator with a median 8.6× speedup (up to 22× on the fastest seeds, under 1000 epochs), growing monotonically as the norm is held lower; the residual stalls also vanish, though significant only pooled over the two mitigations run at both strengths (0/40 vs 6/20, p=7.7×10-4), not per method. The grokking delay is, causally, the time to form the right representational structure -- decided at the level of features, not labels.