# The Weight Norm Sets the Grokking Timescale: A Causal Delay Law

arXiv: https://arxiv.org/abs/2606.13753 · HTML: https://mlsanity.com/paper/2606.13753/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 11 Jun 2026.

## Builds on (most cited first, 22 of 22)

- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit](https://mlsanity.com/paper/2207.08799/) (2022)
- [Explaining grokking through circuit efficiency](https://mlsanity.com/paper/2309.02390/) (2023)
- [Grokking as the Transition from Lazy to Rich Training Dynamics](https://mlsanity.com/paper/2310.06110/) (2023)
- [Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking](https://mlsanity.com/paper/2311.18817/) (2023)
- [Confidence Regulation Neurons in Language Models](https://mlsanity.com/paper/2406.16254/) (2024)
- [Grokking Beyond the Euclidean Norm of Model Parameters](https://mlsanity.com/paper/2506.05718/) (2025)
- [Progress Measures for Grokking on Real-world Tasks](https://mlsanity.com/paper/2405.12755/) (2024)
- [What Can Grokking Teach Us About Learning Under Nonstationarity?](https://mlsanity.com/paper/2507.20057/) (2025)
- [The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization](https://mlsanity.com/paper/2603.13331/) (2026)
- [A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization](https://mlsanity.com/paper/2603.25009/) (2026)
- [The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology](https://mlsanity.com/paper/2603.05228/) (2026)
- [Gated Normalization Removal and Scale Anchoring in Pre-Norm Transformers](https://mlsanity.com/paper/2602.10408/) (2026)
- [Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold](https://mlsanity.com/paper/2604.00230/) (2026)
- [Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics](https://mlsanity.com/paper/2605.20441/) (2026)
- [When Data Falls Short: Grokking Below the Critical Threshold](https://mlsanity.com/paper/2511.04760/) (2025)

## Built on by (most cited first, 4 of 4)

- [Structure-Specific Representational Priors Causally Control the Grokking Delay](https://mlsanity.com/paper/2607.04333/) (2026)
- [What Makes a Representational Prior Work? Feature Families, Label-Free Invariances, and Critical Windows in Grokking](https://mlsanity.com/paper/2607.12735/) (2026)
- [Mapping the Emergence of Regularization-Driven Dynamics in Grokking](https://mlsanity.com/paper/2608.25813/) (2026)
- [Capability Emergence Can Be Forecast: Per-Seed, In Advance, With Calibrated Intervals, Certified False Alarms, and a Blind Pre-Registered Gate](https://mlsanity.com/paper/2609.19000/) (2026)

## Abstract (arXiv)

Grokking is the delayed onset of generalization in neural networks, arising long after they fit the training data. Whether the weight norm causes this delay is disputed: some studies report a critical norm at the transition, others observe grokking with no fixed norm at all. We settle this by intervening on the norm during training rather than only observing it. Under free training with weight decay, networks grok when the weight norm reaches a value Wc that varies little across seeds and learning rates (CV 1 to 2 percent) and grows with the modular base as a power law. When we instead clamp the norm to a fixed multiple rho of Wc and hold it there, the network still groks, but the delay follows T_grok proportional to exp(alpha rho). One exponent, alpha near 7.5, fits this delay across four moduli (R^2 = 0.996). Over the swept ranges the held norm moves the delay by about 19x and the learning rate by only about 2x, and holding the norm above Wc slows grokking rather than preventing it. A final LayerNorm removes the dependence by decoupling weight scale from the network function; without it the exponential law returns. This pinned-norm delay is the exponential counterpart to the logarithmic delay predicted for a freely contracting norm.