mlsanityData to 14 Oct 2026

To Grok Grokking: Provable Grokking in Ridge Regression

arXiv 2601.19791 ↗First posted 27 Jan 2026cs.LG stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

We study grokking, the onset of generalization long after overfitting, in a classical ridge regression setting. We prove end-to-end grokking results for learning over-parameterized linear regression models using gradient descent with weight decay. Specifically, we prove that the following stages occur: (i) the model overfits the training data early during training; (ii) poor generalization persists long after overfitting has manifested; and (iii) the generalization error eventually becomes arbitrarily small. Moreover, we show, both theoretically and empirically, that grokking can be amplified or eliminated in a principled manner through proper hyperparameter tuning. To the best of our knowledge, these are the first rigorous quantitative bounds on the generalization delay (which we refer to as the "grokking time") in terms of training hyperparameters. Lastly, going beyond the linear setting, we empirically demonstrate that our quantitative bounds also capture the behavior of grokking on non-linear neural networks. Our results suggest that grokking is not an inherent failure mode of deep learning, but rather a consequence of specific training conditions, and thus does not require fundamental changes to the model architecture or learning algorithm to avoid.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 37 of its 48 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

37 of 48 references.

References · 37

  1. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  2. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  3. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  4. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  5. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  6. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 144
  7. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  8. 2023Explaining grokking through circuit efficiencycited by 111
  9. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 108
  10. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 94
  11. 2023Grokking modular arithmeticcited by 83
  12. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 83
25 more
  1. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 80
  2. 2024Deep Networks Always Grok and Here is Whycited by 51
  3. 2023Unifying Grokking and Double Descentcited by 50
  4. 2025Grokking at the Edge of Numerical Stabilitycited by 46
  5. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  6. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 36
  7. 2023Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grokcited by 35
  8. 2023Learning the greatest common divisor: explaining transformer predictionscited by 33
  9. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 33
  10. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 31
  11. 2022Grokking phase transitions in learning local rules with gradient descentcited by 28
  12. 2024Emergence in non-neural models: grokking modular arithmetic via average gradient outer productcited by 28
  13. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 25
  14. 2023Grokking in Linear Estimators -- A Solvable Model that Groks without Understandingcited by 25
  15. 2024Critical Data Size of Language Models from a Grokking Perspectivecited by 24
  16. 2024Deep Grokking: Would Deep Neural Networks Generalize Better?cited by 15
  17. 2025Grokking Beyond the Euclidean Norm of Model Parameterscited by 15
  18. 2023Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexitycited by 13
  19. 2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
  20. 2025Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Modelcited by 13
  21. 2025A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisationcited by 13
  22. 2023Understanding Grokking Through A Robustness Viewpointcited by 12
  23. 2024Grokking at the Edge of Linear Separabilitycited by 12
  24. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
  25. 2025Not All Explanations for Deep Learning Phenomena Are Equally Valuablecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnosticscited by 2
  2. 2026Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetriescited by 2
  3. 2026Mapping the Emergence of Regularization-Driven Dynamics in Grokkingcited by 1
  4. 2026A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decaycited by 1
  5. 2026Canonical Regularisation of Wide Feature-Learning Neural Networkscited by 0
  6. 2026Biologically Inspired Mechanisms for Facilitating Grokking in Multilayer Perceptronscited by 0
  7. 2026A Spectral Theory of Grokking: Weight Decay induces Feature Learningcited by 0
  8. 2026Misalignment of Low-Loss Regions Causes Grokkingcited by 0