mlsanityData to 7 Oct 2026

Exploring Grokking: Experimental and Mechanistic Investigations

arXiv 2412.10898 ↗First posted 14 Dec 2024cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

The phenomenon of grokking in over-parameterized neural networks has garnered significant interest. It involves the neural network initially memorizing the training set with zero training error and near-random test error. Subsequent prolonged training leads to a sharp transition from no generalization to perfect generalization. Our study comprises extensive experiments and an exploration of the research behind the mechanism of grokking. Through experiments, we gained insights into its behavior concerning the training data fraction, the model, and the optimization. The mechanism of grokking has been a subject of various viewpoints proposed by researchers, and we introduce some of these perspectives.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 11 of its 13 references that are in the map. First posted on arXiv 14 Dec 2024.

Builds on

11 of 13 references.

References · 11

  1. 2017Decoupled Weight Decay Regularizationcited by 1,722
  2. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  3. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  4. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  5. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
  6. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  7. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  8. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  9. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  10. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 68
  11. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 24

Built on by

Not cited yet.