# Grokking modular arithmetic

arXiv: https://arxiv.org/abs/2301.02679 · HTML: https://mlsanity.com/paper/2301.02679/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 16 references that are in the map.
- First posted on arXiv 6 Jan 2023.

## Builds on (most cited first, 14 of 14)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [The large learning rate phase of deep learning: the catapult mechanism](https://mlsanity.com/paper/2003.02218/) (2020)
- [Feature Learning in Infinite-Width Neural Networks](https://mlsanity.com/paper/2011.14522/) (2020)
- [Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit](https://mlsanity.com/paper/2207.08799/) (2022)
- [The Principles of Deep Learning Theory](https://mlsanity.com/paper/2106.10165/) (2021)
- [Omnigrok: Grokking Beyond Algorithmic Data](https://mlsanity.com/paper/2210.01117/) (2022)
- [The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon](https://mlsanity.com/paper/2206.04817/) (2022)
- [Grokking phase transitions in learning local rules with gradient descent](https://mlsanity.com/paper/2210.15435/) (2022)
- [How to quantify fields or textures? A guide to the scattering transform](https://mlsanity.com/paper/2112.01288/) (2021)
- [Distinct types of eigenvector localization in networks](https://mlsanity.com/paper/1505.06024/) (2015)

## Built on by (most cited first, 30 of 83)

- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [Grokking as the Transition from Lazy to Rich Training Dynamics](https://mlsanity.com/paper/2310.06110/) (2023)
- [Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking](https://mlsanity.com/paper/2311.18817/) (2023)
- [Grokking as a First Order Phase Transition in Two Layer Networks](https://mlsanity.com/paper/2310.03789/) (2023)
- [Grokking at the Edge of Numerical Stability](https://mlsanity.com/paper/2501.04697/) (2025)
- [Grokfast: Accelerated Grokking by Amplifying Slow Gradients](https://mlsanity.com/paper/2405.20233/) (2024)
- [Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks](https://mlsanity.com/paper/2406.02550/) (2024)
- [Why Do You Grok? A Theoretical Analysis of Grokking Modular Addition](https://mlsanity.com/paper/2407.12332/) (2024)
- [Pretraining on the Test Set Is All You Need](https://mlsanity.com/paper/2309.08632/) (2023)
- [Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data](https://mlsanity.com/paper/2310.02541/) (2023)
- [Feature emergence via margin maximization: case studies in algebraic tasks](https://mlsanity.com/paper/2311.07568/) (2023)
- [Approaching Deep Learning through the Spectral Dynamics of Weights](https://mlsanity.com/paper/2408.11804/) (2024)
- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [Emergence in non-neural models: grokking modular arithmetic via average gradient outer product](https://mlsanity.com/paper/2407.20199/) (2024)
- [Grokking Group Multiplication with Cosets](https://mlsanity.com/paper/2312.06581/) (2023)
- [Algorithmic Capabilities of Random Transformers](https://mlsanity.com/paper/2410.04368/) (2024)
- [Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking](https://mlsanity.com/paper/2509.21519/) (2025)
- [Grokking as Compression: A Nonlinear Complexity Perspective](https://mlsanity.com/paper/2310.05918/) (2023)
- [Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomials](https://mlsanity.com/paper/2402.16726/) (2024)
- [Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks](https://mlsanity.com/paper/2506.06489/) (2025)
- [Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Models](https://mlsanity.com/paper/2411.02083/) (2024)
- [Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild](https://mlsanity.com/paper/2503.10065/) (2025)
- [Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Model](https://mlsanity.com/paper/2504.13292/) (2025)
- [A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisation](https://mlsanity.com/paper/2505.20172/) (2025)
- [Understanding Grokking Through A Robustness Viewpoint](https://mlsanity.com/paper/2311.06597/) (2023)
- [Grokking at the Edge of Linear Separability](https://mlsanity.com/paper/2410.04489/) (2024)
- [Sequential Group Composition: A Window into the Mechanics of Deep Learning](https://mlsanity.com/paper/2602.03655/) (2026)
- [Bayesian RG Flow in Neural Network Field Theories](https://mlsanity.com/paper/2405.17538/) (2024)
- [Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)](https://mlsanity.com/paper/2502.21009/) (2025)

## Abstract (arXiv)

We present a simple neural network that can learn modular arithmetic tasks and exhibits a sudden jump in generalization known as ``grokking''. Concretely, we present (i) fully-connected two-layer networks that exhibit grokking on various modular arithmetic tasks under vanilla gradient descent with the MSE loss function in the absence of any regularization; (ii) evidence that grokking modular arithmetic corresponds to learning specific feature maps whose structure is determined by the task; (iii) analytic expressions for the weights -- and thus for the feature maps -- that solve a large class of modular arithmetic tasks; and (iv) evidence that these feature maps are also found by vanilla gradient descent as well as AdamW, thereby establishing complete interpretability of the representations learnt by the network.