mlsanityData to 14 Oct 2026

Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexity

arXiv 2310.17247 ↗First posted 26 Oct 2023cs.LG stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

In some settings neural networks exhibit a phenomenon known as textitgrokking, where they achieve perfect or near-perfect accuracy on the validation set long after the same performance has been achieved on the training set. In this paper, we discover that grokking is not limited to neural networks but occurs in other settings such as Gaussian process (GP) classification, GP regression, linear regression and Bayesian neural networks. We also uncover a mechanism by which to induce grokking on algorithmic datasets via the addition of dimensions containing spurious information. The presence of the phenomenon in non-neural architectures shows that grokking is not restricted to settings considered in current theoretical and empirical studies. Instead, grokking may be possible in any model where solution search is guided by complexity and error.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 14 of its 35 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

14 of 35 references.

References · 14

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  3. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 609
  4. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  5. 2014Scalable Variational Gaussian Process Classificationcited by 228
  6. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  7. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 146
  8. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 83
  9. 2016Understanding Probabilistic Sparse Gaussian Process Approximationscited by 68
  10. 2021Model Complexity of Deep Learning: A Surveycited by 65
  11. 2023Grokking of Hierarchical Structure in Vanilla Transformerscited by 53
  12. 2022Grokking phase transitions in learning local rules with gradient descentcited by 28
2 more
  1. 2023Grokking in Linear Estimators -- A Solvable Model that Groks without Understandingcited by 25
  2. 2023Grokking as Compression: A Nonlinear Complexity Perspectivecited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 3; 2026: 4.

13 papers.

Sort:

By area:

  1. 2025Deep Learning is Not So Mysterious or Differentcited by 56
  2. 2024Emergence in non-neural models: grokking modular arithmetic via average gradient outer productcited by 28
  3. 2024Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transitioncited by 17
  4. 2024Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomialscited by 14
  5. 2025Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Modelcited by 13
  6. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
  7. 2026To Grok Grokking: Provable Grokking in Ridge Regressioncited by 8
  8. 2023Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networkscited by 6
  9. 2024Language Models "Grok" to Copycited by 5
  10. 2026A Basin-Selection Perspective on Grokking via Singular Learning Theorycited by 4
  11. 2025Not All Explanations for Deep Learning Phenomena Are Equally Valuablecited by 3
  12. 2026Grokking From Abstraction to Intelligencecited by 1
1 more
  1. 2026A Spectral Theory of Grokking: Weight Decay induces Feature Learningcited by 0