mlsanityData to 7 Oct 2026

Towards Understanding Grokking: An Effective Theory of Representation Learning

arXiv 2205.10343 ↗First posted 20 May 2022cs.LG cond-mat.dis-nn cond-mat.stat-mechcited by 229

Not on the map. Why?

Abstract (arXiv)

We aim to understand grokking, a phenomenon where models generalize long after overfitting their training set. We present both a microscopic analysis anchored by an effective theory and a macroscopic analysis of phase diagrams describing learning performance across hyperparameters. We find that generalization originates from structured representations whose training dynamics and dependence on training set size can be predicted by our effective theory in a toy setting. We observe empirically the presence of four learning phases: comprehension, grokking, memorization, and confusion. We find representation learning to occur only in a "Goldilocks zone" (including comprehension and grokking) between memorization and confusion. We find on transformers the grokking phase stays closer to the memorization phase (compared to the comprehension phase), leading to delayed generalization. The Goldilocks phase is reminiscent of "intelligence from starvation" in Darwinian evolution, where resource limitations drive discovery of more efficient solutions. This study not only provides intuitive explanations of the origin of grokking, but also highlights the usefulness of physics-inspired tools, e.g., effective theories and phase diagrams, for understanding deep learning.

Key facts. Cited by 229 papers in the map (data through 7 Oct 2026). Builds on 23 of its 35 references that are in the map. First posted on arXiv 20 May 2022.

Builds on

23 of 35 references.

References · 23

  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2017Deep Setscited by 1,394
  4. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  5. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  6. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  7. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
  8. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  9. 2022Predictability and Surprise in Large Generative Modelscited by 209
  10. 2018Training behavior of deep neural network in frequency domaincited by 174
  11. 2021The Principles of Deep Learning Theorycited by 169
  12. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
11 more
  1. 2020An Overview of Deep Semi-Supervised Learningcited by 121
  2. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  3. 2020Optimal Regularization Can Mitigate Double Descentcited by 107
  4. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  5. 2020Neural Networks and Quantum Field Theorycited by 58
  6. 2020Learning to Unknotcited by 40
  7. 2019A type of generalization error induced by initialization in deep neural networkscited by 33
  8. 2021Machine-Learning Mathematical Structurescited by 25
  9. 2021Multi-scale Feature Learning Dynamics: Insights for Double Descentcited by 24
  10. 2020A Free-Energy Principle for Representation Learningcited by 7
  11. 2015Visual Learning of Arithmetic Operationscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 34; 2024: 63; 2025: 60; 2026: 68.

229 papers.

Sort:

By area:

  1. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  2. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  3. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  4. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  5. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  6. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  7. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  8. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  9. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  10. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  11. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  12. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
38 more
  1. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  2. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  3. 2023Explaining grokking through circuit efficiencycited by 105
  4. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  5. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  6. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  7. 2023Grokking modular arithmeticcited by 77
  8. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  9. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
  10. 2023Unifying Grokking and Double Descentcited by 49
  11. 2024ICLR: In-Context Learning of Representationscited by 49
  12. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  13. 2025The Origins of Representation Manifolds in Large Language Modelscited by 46
  14. 2023Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretabilitycited by 38
  15. 2024Grokfast: Accelerated Grokking by Amplifying Slow Gradientscited by 38
  16. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 34
  17. 2023Pretraining on the Test Set Is All You Needcited by 33
  18. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  19. 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33
  20. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
  21. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 32
  22. 2023Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grokcited by 31
  23. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  24. 2025Mechanistic understanding and validation of large AI models with SemanticLenscited by 29
  25. 2022Grokking phase transitions in learning local rules with gradient descentcited by 28
  26. 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 28
  27. 2022Why Random Pruning Is All We Need to Start Sparsecited by 26
  28. 2024Emergence in non-neural models: grokking modular arithmetic via average gradient outer productcited by 26
  29. 2023Grokking in Linear Estimators -- A Solvable Model that Groks without Understandingcited by 25
  30. 2024Zero-shot forecasting of chaotic systemscited by 25
  31. 2023To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasetscited by 24
  32. 2023Interpretability Illusions in the Generalization of Simplified Modelscited by 24
  33. 2024Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformerscited by 24
  34. 2024Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competitioncited by 23
  35. 2024Critical Data Size of Language Models from a Grokking Perspectivecited by 22
  36. 2024Towards Uncovering How Large Language Model Works: An Explainability Perspectivecited by 22
  37. 2024The Complexity Dynamics of Grokkingcited by 21
  38. 2025Early Stopping Against Label Noise Without Validation Datacited by 21

50 most cited of 229 All 229 (JSON)