mlsanityData to 7 Oct 2026

Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding

arXiv 2310.16441 ↗First posted 25 Oct 2023stat.ML cond-mat.dis-nn cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Grokking is the intriguing phenomenon where a model learns to generalize long after it has fit the training data. We show both analytically and numerically that grokking can surprisingly occur in linear networks performing linear tasks in a simple teacher-student setup with Gaussian inputs. In this setting, the full training dynamics is derived in terms of the training and generalization data covariance matrix. We present exact predictions on how the grokking time depends on input and output dimensionality, train sample size, regularization, and network initialization. We demonstrate that the sharp increase in generalization accuracy may not imply a transition from "memorization" to "understanding", but can simply be an artifact of the accuracy measure. We provide empirical verification for our calculations, along with preliminary results indicating that some predictions also hold for deeper networks, with non-linear activations.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 14 of its 21 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

14 of 21 references.

References · 14

  1. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  3. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  4. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  5. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  6. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  7. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  8. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  9. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  10. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 78
  11. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 61
  12. 2023Unifying Grokking and Double Descentcited by 49
2 more
  1. 2023Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grokcited by 31
  2. 2022Gradient flow in the gaussian covariate model: exact solution of learning curves and multiple descent structurescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 8; 2025: 6; 2026: 9.

25 papers.

Sort:

By area:

  1. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  2. 2025Deep Learning is Not So Mysterious or Differentcited by 55
  3. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 34
  4. 2024Towards Uncovering How Large Language Model Works: An Explainability Perspectivecited by 22
  5. 2024Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomialscited by 14
  6. 2023Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexitycited by 13
  7. 2024Grokking at the Edge of Linear Separabilitycited by 11
  8. 2024On Regularization via Early Stopping for Least Squares Regressioncited by 10
  9. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
  10. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 8
  11. 2024Phase Transitions in the Output Distribution of Large Language Modelscited by 7
  12. 2026To Grok Grokking: Provable Grokking in Ridge Regressioncited by 6
13 more
  1. 2026Noise-Driven Escape from Metastable Phases explains Grokking in Deep Neural Networkscited by 5
  2. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
  3. 2026Statistical Properties of Training & Generalizationcited by 2
  4. 2024Measuring Sharpness in Grokkingcited by 1
  5. 2025Grokking Explained: A Statistical Phenomenoncited by 1
  6. 2025Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokkingcited by 1
  7. 2025When Data Falls Short: Grokking Below the Critical Thresholdcited by 1
  8. 2026The Implicit Bias of Logit Regularizationcited by 1
  9. 2026Grokking Finite-Dimensional Algebracited by 1
  10. 2026Grokking in Linear Models for Logistic Regressioncited by 0
  11. 2026A Bayesian Perspective on the Role of Epistemic Uncertainty for Delayed Generalization in In-Context Learningcited by 0
  12. 2026Structured Features Overfit Where Random Features Grokcited by 0
  13. 2026A Spectral Theory of Grokking: Weight Decay induces Feature Learningcited by 0