mlsanityData to 7 Oct 2026

Grokking modular arithmetic

arXiv 2301.02679 ↗First posted 6 Jan 2023cs.LG cond-mat.dis-nncited by 77

Not on the map. Why?

Abstract (arXiv)

We present a simple neural network that can learn modular arithmetic tasks and exhibits a sudden jump in generalization known as ``grokking''. Concretely, we present (i) fully-connected two-layer networks that exhibit grokking on various modular arithmetic tasks under vanilla gradient descent with the MSE loss function in the absence of any regularization; (ii) evidence that grokking modular arithmetic corresponds to learning specific feature maps whose structure is determined by the task; (iii) analytic expressions for the weights -- and thus for the feature maps -- that solve a large class of modular arithmetic tasks; and (iv) evidence that these feature maps are also found by vanilla gradient descent as well as AdamW, thereby establishing complete interpretability of the representations learnt by the network.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 14 of its 16 references that are in the map. First posted on arXiv 6 Jan 2023.

Builds on

14 of 16 references.

References · 14

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  3. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  4. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  5. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  6. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  7. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  8. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  9. 2021The Principles of Deep Learning Theorycited by 169
  10. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  11. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  12. 2022Grokking phase transitions in learning local rules with gradient descentcited by 28
2 more
  1. 2021How to quantify fields or textures? A guide to the scattering transformcited by 5
  2. 2015Distinct types of eigenvector localization in networkscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 15; 2025: 23; 2026: 25.

77 papers.

Sort:

By area:

  1. 2023Large Language Modelscited by 774
  2. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  3. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  4. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  5. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  6. 2025Grokking at the Edge of Numerical Stabilitycited by 41
  7. 2024Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic taskscited by 40
  8. 2024Grokfast: Accelerated Grokking by Amplifying Slow Gradientscited by 38
  9. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 34
  10. 2023Pretraining on the Test Set Is All You Needcited by 33
  11. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  12. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 32
38 more
  1. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  2. 2024Emergence in non-neural models: grokking modular arithmetic via average gradient outer productcited by 26
  3. 2023Grokking Group Multiplication with Cosetscited by 22
  4. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  5. 2024Algorithmic Capabilities of Random Transformerscited by 17
  6. 2023Grokking as Compression: A Nonlinear Complexity Perspectivecited by 14
  7. 2024Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomialscited by 14
  8. 2025Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokkingcited by 14
  9. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  10. 2025Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Modelcited by 13
  11. 2024Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Modelscited by 12
  12. 2024Grokking at the Edge of Linear Separabilitycited by 11
  13. 2025A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisationcited by 11
  14. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  15. 2023Understanding Grokking Through A Robustness Viewpointcited by 10
  16. 2024Bayesian RG Flow in Neural Network Field Theoriescited by 9
  17. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 8
  18. 2025Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokkingcited by 8
  19. 2025The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifoldcited by 8
  20. 2023Learning to Compute Gr"obner Basescited by 7
  21. 2024Salsa Fresca: Angular Embeddings and Pre-Training for ML Attacks on Learning With Errorscited by 7
  22. 2026Sequential Group Composition: A Window into the Mechanics of Deep Learningcited by 7
  23. 2025Mechanistic Insights into Grokking from the Embedding Layercited by 6
  24. 2026To Grok Grokking: Provable Grokking in Ridge Regressioncited by 6
  25. 2023Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networkscited by 5
  26. 2024Clustering and Alignment: Understanding the Training Dynamics in Modular Additioncited by 5
  27. 2024Composing Global Solutions to Reasoning Tasks via Algebraic Objects in Neural Netscited by 5
  28. 2026Convergent Evolution: How Different Language Models Learn Similar Number Representationscited by 5
  29. 2024Making Hard Problems Easier with Custom Data Distributions and Loss Regularization: A Case Study in Modular Arithmeticcited by 4
  30. 2025Feature Identification via the Empirical NTKcited by 4
  31. 2026Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikescited by 4
  32. 2025Learning words in groups: fusion algebras, tensor ranks and grokkingcited by 3
  33. 2025On the Emergence of Induction Heads for In-Context Learningcited by 3
  34. 2025On the geometry and topology of representations: the manifolds of modular additioncited by 3
  35. 2026Structure-Specific Representational Priors Causally Control the Grokking Delaycited by 3
  36. 2023Gradient Descent Fails to Learn High-frequency Functions and Modular Arithmeticcited by 2
  37. 2024Machine learning for modular multiplicationcited by 2
  38. 2025The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representationscited by 2

50 most cited of 77 All 77 (JSON)