# Grokking Beyond the Euclidean Norm of Model Parameters

arXiv: https://arxiv.org/abs/2506.05718 · HTML: https://mlsanity.com/paper/2506.05718/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2025.

## Built on by (most cited first, 15 of 15)

- [To Grok Grokking: Provable Grokking in Ridge Regression](https://mlsanity.com/paper/2601.19791/) (2026)
- [The Weight Norm Sets the Grokking Timescale: A Causal Delay Law](https://mlsanity.com/paper/2606.13753/) (2026)
- [Grokking as a Falsifiable Finite-Size Transition](https://mlsanity.com/paper/2603.24746/) (2026)
- [Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokking](https://mlsanity.com/paper/2510.04930/) (2025)
- [Grokking Finite-Dimensional Algebra](https://mlsanity.com/paper/2602.19533/) (2026)
- [Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetries](https://mlsanity.com/paper/2607.23967/) (2026)
- [Implicitly Normalized Online PCA: A Regularized Algorithm with Exact High-Dimensional Dynamics](https://mlsanity.com/paper/2512.01231/) (2025)
- [On the Dynamics of Delayed Generalization in Linear Models](https://mlsanity.com/paper/2602.08302/) (2026)
- [Grokking From Abstraction to Intelligence](https://mlsanity.com/paper/2603.29262/) (2026)
- [Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorize](https://mlsanity.com/paper/2605.04396/) (2026)
- [Grokking as Structural Inference: Transformers Need Bayesian Lottery Tickets](https://mlsanity.com/paper/2605.15787/) (2026)
- [What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy](https://mlsanity.com/paper/2606.18465/) (2026)
- [Mapping the Emergence of Regularization-Driven Dynamics in Grokking](https://mlsanity.com/paper/2608.25813/) (2026)
- [A Spectral Theory of Grokking: Weight Decay induces Feature Learning](https://mlsanity.com/paper/2609.26679/) (2026)
- [Statistical Vignettes on the Grokking Phenomenon](https://mlsanity.com/paper/2610.18811/) (2026)

## Abstract (arXiv)

Grokking refers to a delayed generalization following overfitting when optimizing artificial neural networks with gradient-based methods. In this work, we demonstrate that grokking can be induced by regularization, either explicit or implicit. More precisely, we show that when there exists a model with a property P (e.g., sparse or low-rank weights) that generalizes on the problem of interest, gradient descent with a small but non-zero regularization of P (e.g., ℓ1 or nuclear norm regularization) results in grokking. This extends previous work showing that small non-zero weight decay induces grokking. Moreover, our analysis shows that over-parameterization by adding depth makes it possible to grok or ungrok without explicitly using regularization, which is impossible in shallow cases. We further show that the ℓ2 norm is not a reliable proxy for generalization when the model is regularized toward a different property P, as the ℓ2 norm grows in many cases where no weight decay is used, but the model generalizes anyway. We also show that grokking can be amplified solely through data selection, with any other hyperparameter fixed.