# To Grok Grokking: Provable Grokking in Ridge Regression arXiv: https://arxiv.org/abs/2601.19791 ยท HTML: https://mlsanity.com/paper/2601.19791/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 37 of its 48 references that are in the map. - First posted on arXiv 27 Jan 2026. ## Builds on (most cited first, 30 of 37) - [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023) - [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022) - [Characterizing Implicit Bias in Terms of Optimization Geometry](https://mlsanity.com/paper/1802.08246/) (2018) - [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022) - [Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit](https://mlsanity.com/paper/2207.08799/) (2022) - [A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations](https://mlsanity.com/paper/2302.03025/) (2023) - [Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process](https://mlsanity.com/paper/1904.09080/) (2019) - [Explaining grokking through circuit efficiency](https://mlsanity.com/paper/2309.02390/) (2023) - [Grokking as the Transition from Lazy to Rich Training Dynamics](https://mlsanity.com/paper/2310.06110/) (2023) - [The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon](https://mlsanity.com/paper/2206.04817/) (2022) - [Grokking modular arithmetic](https://mlsanity.com/paper/2301.02679/) (2023) - [A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworks](https://mlsanity.com/paper/2303.11873/) (2023) - [Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking](https://mlsanity.com/paper/2311.18817/) (2023) - [Deep Networks Always Grok and Here is Why](https://mlsanity.com/paper/2402.15555/) (2024) - [Unifying Grokking and Double Descent](https://mlsanity.com/paper/2303.06173/) (2023) - [Grokking at the Edge of Numerical Stability](https://mlsanity.com/paper/2501.04697/) (2025) - [Implicit Regularization in ReLU Networks with the Square Loss](https://mlsanity.com/paper/2012.05156/) (2020) - [Why Do You Grok? A Theoretical Analysis of Grokking Modular Addition](https://mlsanity.com/paper/2407.12332/) (2024) - [Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok](https://mlsanity.com/paper/2306.13253/) (2023) - [Learning the greatest common divisor: explaining transformer predictions](https://mlsanity.com/paper/2308.15594/) (2023) - [Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data](https://mlsanity.com/paper/2310.02541/) (2023) - [Approaching Deep Learning through the Spectral Dynamics of Weights](https://mlsanity.com/paper/2408.11804/) (2024) - [Grokking phase transitions in learning local rules with gradient descent](https://mlsanity.com/paper/2210.15435/) (2022) - [Emergence in non-neural models: grokking modular arithmetic via average gradient outer product](https://mlsanity.com/paper/2407.20199/) (2024) - [Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn features](https://mlsanity.com/paper/2212.13881/) (2022) - [Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding](https://mlsanity.com/paper/2310.16441/) (2023) - [Critical Data Size of Language Models from a Grokking Perspective](https://mlsanity.com/paper/2401.10463/) (2024) - [Deep Grokking: Would Deep Neural Networks Generalize Better?](https://mlsanity.com/paper/2405.19454/) (2024) - [Grokking Beyond the Euclidean Norm of Model Parameters](https://mlsanity.com/paper/2506.05718/) (2025) - [Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexity](https://mlsanity.com/paper/2310.17247/) (2023) ## Built on by (most cited first, 8 of 8) - [Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics](https://mlsanity.com/paper/2605.20441/) (2026) - [Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetries](https://mlsanity.com/paper/2607.23967/) (2026) - [Mapping the Emergence of Regularization-Driven Dynamics in Grokking](https://mlsanity.com/paper/2608.25813/) (2026) - [A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay](https://mlsanity.com/paper/2609.07755/) (2026) - [Canonical Regularisation of Wide Feature-Learning Neural Networks](https://mlsanity.com/paper/2605.18180/) (2026) - [Biologically Inspired Mechanisms for Facilitating Grokking in Multilayer Perceptrons](https://mlsanity.com/paper/2608.28184/) (2026) - [A Spectral Theory of Grokking: Weight Decay induces Feature Learning](https://mlsanity.com/paper/2609.26679/) (2026) - [Misalignment of Low-Loss Regions Causes Grokking](https://mlsanity.com/paper/2610.00620/) (2026) ## Abstract (arXiv) We study grokking, the onset of generalization long after overfitting, in a classical ridge regression setting. We prove end-to-end grokking results for learning over-parameterized linear regression models using gradient descent with weight decay. Specifically, we prove that the following stages occur: (i) the model overfits the training data early during training; (ii) poor generalization persists long after overfitting has manifested; and (iii) the generalization error eventually becomes arbitrarily small. Moreover, we show, both theoretically and empirically, that grokking can be amplified or eliminated in a principled manner through proper hyperparameter tuning. To the best of our knowledge, these are the first rigorous quantitative bounds on the generalization delay (which we refer to as the "grokking time") in terms of training hyperparameters. Lastly, going beyond the linear setting, we empirically demonstrate that our quantitative bounds also capture the behavior of grokking on non-linear neural networks. Our results suggest that grokking is not an inherent failure mode of deep learning, but rather a consequence of specific training conditions, and thus does not require fundamental changes to the model architecture or learning algorithm to avoid.