First-Passage Prediction of Grokking Delay: ACalibrated Law under AdamW with Causal Validation
Not on the map. Why?
Abstract (arXiv)
We give the first quantitative prediction of grokking delay under AdamW. Treating the delay as a first-passage time, we derive a closed-form law T_grok - T_mem = (1 / 2 kappa_LL eta lambda) log(V_mem / V_star), where V_t = ||theta_t||^2 is the squared parameter norm, V_star is an architecture-dependent threshold, and kappa_LL absorbs the AdamW correction to the clean-SGD contraction rate 2 eta lambda. Calibrating (kappa_LL, V_star) on a single hyperparameter cell predicts grokking delays on 26 held-out runs with MAPE 17.7% over a 41x delay range; the law generalises to MLPs (MAPE 18.0%, N=34) and degrades to 23.3% on cross-task extension (N=46, 43.5x range), with a structured residual in which V_star / V_mem stays comparatively stable within architecture (CV about 14% on the 1L transformer). First-passage of V_t is necessary but not sufficient. A quantile-margin theorem establishes that positive delay requires both norm separation V_mem > V_post and angular reachability of a threshold alpha_star = arcsin(C / V_T_mem^(1/2)), where C is computable from the empirical NTK feature map and the validation-margin quantile. Calibrating C on modulus p=89 predicts alpha_star = 47.2 degrees at p=97 (observed 47.8 degrees, error 1.3%) as a prior cross-cell prediction. Causal interventions that freeze the norm or remove weight decay at memorisation eliminate grokking (0/6 vs. 3/3 baseline), trapping the angular displacement near 12 degrees. kappa_LL is empirically measured per architecture rather than derived from (beta_1, beta_2, epsilon); within-architecture CV stays at most 15% across four architectures, but values differ by about 2x between architectural variants beyond depth alone. Empirical scope is algorithmic tasks (modular arithmetic, sparse parity) under AdamW; whether the law transfers to natural-language scale models is open.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 29 references that are in the map. First posted on arXiv 13 May 2026.
Builds on
29 of 29 references.
References · 29
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2017Decoupled Weight Decay Regularizationcited by 1,831
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
- 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
- 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
- 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
- 2020Directional convergence and alignment in deep learningcited by 178
- 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 146
17 more
- 2023Explaining grokking through circuit efficiencycited by 111
- 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 108
- 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
- 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 83
- 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 80
- 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
- 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
- 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 53
- 2025Grokking at the Edge of Numerical Stabilitycited by 46
- 2024The Implicit Bias of Adam on Separable Datacited by 36
- 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 36
- 2023Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grokcited by 35
- 2025A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisationcited by 13
- 2025The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifoldcited by 9
- 2026The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalizationcited by 7
- 2020Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGDcited by 6
- 2026The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topologycited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.