mlsanityData to 7 Oct 2026

Why Do You Grok? A Theoretical Analysis of Grokking Modular Addition

arXiv 2407.12332 ↗First posted 17 Jul 2024cs.LG stat.MLcited by 34

Not on the map. Why?

Abstract (arXiv)

We present a theoretical explanation of the ``grokking'' phenomenon, where a model generalizes long after overfitting,for the originally-studied problem of modular addition. First, we show that early in gradient descent, when the ``kernel regime'' approximately holds, no permutation-equivariant model can achieve small population error on modular addition unless it sees at least a constant fraction of all possible data points. Eventually, however, models escape the kernel regime. We show that two-layer quadratic networks that achieve zero training loss with bounded ℓ∞ norm generalize well with substantially fewer training points, and further show such networks exist and can be found by gradient descent with small ℓ∞ regularization. We further provide empirical evidence that these networks as well as simple Transformers, leave the kernel regime only after initially overfitting. Taken together, our results strongly support the case for grokking as a consequence of the transition from kernel-like behavior to limiting behavior of gradient descent on deep networks.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 43 of its 55 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

43 of 55 references.

References · 43

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  5. 2018On Lazy Training in Differentiable Programmingcited by 671
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  7. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  9. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  10. 2017Implicit Regularization in Matrix Factorizationcited by 318
  11. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  12. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
31 more
  1. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  2. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  3. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  4. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  5. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  6. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  7. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  8. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  9. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  10. 2023Explaining grokking through circuit efficiencycited by 105
  11. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  12. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  13. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  14. 2023Grokking modular arithmeticcited by 77
  15. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  16. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  17. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 68
  18. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  19. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  20. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  21. 2023Learning the greatest common divisor: explaining transformer predictionscited by 33
  22. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  23. 2020Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets?cited by 32
  24. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 32
  25. 2024The Implicit Bias of Adam on Separable Datacited by 32
  26. 2023Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grokcited by 31
  27. 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
  28. 2023Grokking in Linear Estimators -- A Solvable Model that Groks without Understandingcited by 25
  29. 2022On the non-universality of deep learning: quantifying the cost of symmetrycited by 17
  30. 2022Feature selection with gradient descent on two-layer networks in low-rotation regimescited by 13
  31. 2020When Hardness of Approximation Meets Hardness of Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 18.

34 papers.

Sort:

By area:

  1. 2024Emergence in non-neural models: grokking modular arithmetic via average gradient outer productcited by 26
  2. 2025Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokkingcited by 14
  3. 2025Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Modelcited by 13
  4. 2025A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisationcited by 11
  5. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 8
  6. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 8
  7. 2025Mechanistic Insights into Grokking from the Embedding Layercited by 6
  8. 2025On Learning Verifiers and Implications to Chain-of-Thought Reasoningcited by 6
  9. 2026To Grok Grokking: Provable Grokking in Ridge Regressioncited by 6
  10. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 6
  11. 2024Making Hard Problems Easier with Custom Data Distributions and Loss Regularization: A Case Study in Modular Arithmeticcited by 4
  12. 2026A Basin-Selection Perspective on Grokking via Singular Learning Theorycited by 4
22 more
  1. 2026Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speedscited by 4
  2. 2025Learning words in groups: fusion algebras, tensor ranks and grokkingcited by 3
  3. 2025Grokking vs. Learning: Same Features, Different Encodingscited by 2
  4. 2026Topological Signatures of Grokkingcited by 2
  5. 2026Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmeticcited by 2
  6. 2026Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetriescited by 2
  7. 2025Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokkingcited by 1
  8. 2025A Two-Phase Perspective on Deep Learning Dynamicscited by 1
  9. 2025Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokkingcited by 1
  10. 2026Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutionscited by 1
  11. 2026Breaking Data Symmetry is Needed For Generalization in Feature Learning Kernelscited by 1
  12. 2026Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reductioncited by 1
  13. 2024Do Mice Grok? Glimpses of Hidden Progress During Overtraining in Sensory Cortexcited by 0
  14. 2025Low Rank and Sparse Fourier Structure in Recurrent Networks Trained on Modular Additioncited by 0
  15. 2025Tracing the Path to Grokking: Embeddings, Dropout, and Network Activationcited by 0
  16. 2026Grokking in Linear Models for Logistic Regressioncited by 0
  17. 2026A Bayesian Perspective on the Role of Epistemic Uncertainty for Delayed Generalization in In-Context Learningcited by 0
  18. 2026Learning Large-Scale Modular Addition with an Auxiliary Moduluscited by 0
  19. 2026A Function-Space Approach to the Statistical Mechanics of Learning Dynamicscited by 0
  20. 2026Structured Features Overfit Where Random Features Grokcited by 0
  21. 2026Where Grokking Happens: Distributed Utility and Fourier Recoding Without a Module Switchcited by 0
  22. 2026A Spectral Theory of Grokking: Weight Decay induces Feature Learningcited by 0