# Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetries

arXiv: https://arxiv.org/abs/2607.23967 · HTML: https://mlsanity.com/paper/2607.23967/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 27 Jul 2026.

## Builds on (most cited first, 18 of 18)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced](https://mlsanity.com/paper/1806.00900/) (2018)
- [Omnigrok: Grokking Beyond Algorithmic Data](https://mlsanity.com/paper/2210.01117/) (2022)
- [Grokking as the Transition from Lazy to Rich Training Dynamics](https://mlsanity.com/paper/2310.06110/) (2023)
- [Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking](https://mlsanity.com/paper/2311.18817/) (2023)
- [Deep Networks Always Grok and Here is Why](https://mlsanity.com/paper/2402.15555/) (2024)
- [Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks](https://mlsanity.com/paper/2105.02716/) (2021)
- [Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows](https://mlsanity.com/paper/2307.00144/) (2023)
- [Why Do You Grok? A Theoretical Analysis of Grokking Modular Addition](https://mlsanity.com/paper/2407.12332/) (2024)
- [Grokking Beyond the Euclidean Norm of Model Parameters](https://mlsanity.com/paper/2506.05718/) (2025)
- [A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisation](https://mlsanity.com/paper/2505.20172/) (2025)
- [Grokking at the Edge of Linear Separability](https://mlsanity.com/paper/2410.04489/) (2024)
- [Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows](https://mlsanity.com/paper/2405.12888/) (2024)
- [To Grok Grokking: Provable Grokking in Ridge Regression](https://mlsanity.com/paper/2601.19791/) (2026)
- [The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization](https://mlsanity.com/paper/2603.13331/) (2026)

## Built on by (most cited first, 2 of 2)

- [Small Enough to Know Everything: The Fully-Enumerable Transformer as an Instrument for the Science of Delayed Generalization](https://mlsanity.com/paper/2609.20166/) (2026)
- [A Spectral Theory of Grokking: Weight Decay induces Feature Learning](https://mlsanity.com/paper/2609.26679/) (2026)

## Abstract (arXiv)

Delayed generalization, or grokking, remains poorly understood despite extensive empirical study. We identify an exactly solvable late-time relaxation mechanism for grokking in linear models trained with full-batch heavy-ball optimization and weight decay, together with a locally quadratic extension to nonlinear neural networks. Our analysis reveals a distinguished population-active component of the empirical null space, which we call the grokking subspace. Along this subspace, the training predictions remain unchanged, leaving weight decay as the sole restoring force and giving rise to a slow dissipative relaxation governed by an exact discrete-time and continuous-time law. We show that only this subspace contributes to the slow asymptotic decay of the population risk and derive explicit iteration-scale predictions for the grokking time, recovering the familiar (1-β)/(ηλ) scaling in the weak-regularization regime. The theory further predicts distinct effects of optimizer choice, distinguishing coupled L2 regularization from decoupled weight decay, and yields causal predictions for interventions that modify the grokking component. We verify all theoretical identities without fitted parameters in a synthetic model where every subspace and relaxation rate is computable in closed form. We further observe genuine delayed generalization in modular addition, where the measured delay follows the predicted scaling and the late-time relaxation agrees closely with the theoretical clock.