mlsanityData to 7 Oct 2026

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

arXiv 2201.02177 ↗First posted 6 Jan 2022cs.LGcited by 599

Not on the map. Why?

Cited by 46 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper we propose to study generalization of neural networks on small algorithmically generated datasets. In this setting, questions about data efficiency, memorization, generalization, and speed of learning can be studied in great detail. In some situations we show that neural networks learn through a process of "grokking" a pattern in the data, improving generalization performance from random chance level to perfect generalization, and that this improvement in generalization can happen well past the point of overfitting. We also study generalization as a function of dataset size and find that smaller datasets require increasing amounts of optimization for generalization. We argue that these datasets provide a fertile ground for studying a poorly understood aspect of deep learning: generalization of overparametrized neural networks beyond memorization of the finite training dataset.

Key facts. Cited by 599 papers in the map (data through 7 Oct 2026). Builds on 16 of its 22 references that are in the map. First posted on arXiv 6 Jan 2022.

Builds on

16 of 22 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  5. 2014Neural Turing Machinescited by 630
  6. 2018Universal Transformerscited by 613
  7. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  8. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  9. 2014Memory Networkscited by 354
  10. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  11. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  12. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
4 more
  1. 2015Neural GPUs Learn Algorithmscited by 118
  2. 2015Neural Programmer-Interpreterscited by 96
  3. 2020Triple descent and the two kinds of overfitting: Where & why do they appear?cited by 64
  4. 2015Learning to Transduce with Unbounded Memorycited by 54

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 34; 2023: 76; 2024: 104; 2025: 154; 2026: 231.

599 papers.

Sort:

By area:

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2022In-context Learning and Induction Headscited by 930
  3. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  4. 2023Large Language Modelscited by 774
  5. 2022Transformers learn in-context by gradient descentcited by 714
  6. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  7. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  8. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  9. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  10. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  11. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  12. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
38 more
  1. 2022Neural Networks and the Chomsky Hierarchycited by 214
  2. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  3. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  4. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  5. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  6. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  7. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  8. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  9. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  10. 2023The Quantization Model of Neural Scalingcited by 137
  11. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  12. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  13. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  14. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  15. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  16. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 108
  17. 2023Explaining grokking through circuit efficiencycited by 105
  18. 2022Broken Neural Scaling Lawscited by 102
  19. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  20. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  21. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  22. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  23. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  24. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  25. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  26. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 86
  27. 2025Model Organisms for Emergent Misalignmentcited by 84
  28. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  29. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 82
  30. 2022Training Trajectories of Language Models Across Scalescited by 80
  31. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 78
  32. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  33. 2023Grokking modular arithmeticcited by 77
  34. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  35. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 72
  36. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 68
  37. 2023Length Generalization in Arithmetic Transformerscited by 64
  38. 2022X-Risk Analysis for AI Researchcited by 60

50 most cited of 599 All 599 (JSON)