# Unifying Grokking and Double Descent

arXiv: https://arxiv.org/abs/2303.06173 · HTML: https://mlsanity.com/paper/2303.06173/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 10 Mar 2023.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Optimal Policies Tend to Seek Power](https://mlsanity.com/paper/1912.01683/) (2019)
- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [Is Power-Seeking AI an Existential Risk?](https://mlsanity.com/paper/2206.13353/) (2022)
- [Early Stopping in Deep Networks: Double Descent and How to Eliminate it](https://mlsanity.com/paper/2007.10099/) (2020)
- [Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics](https://mlsanity.com/paper/2209.10015/) (2022)
- [Multi-scale Feature Learning Dynamics: Insights for Double Descent](https://mlsanity.com/paper/2112.03215/) (2021)
- [When and how epochwise double descent happens](https://mlsanity.com/paper/2108.12006/) (2021)

## Built on by (most cited first, 30 of 50)

- [CycleResearcher: Improving Automated Research via Automated Review](https://mlsanity.com/paper/2411.00816/) (2024)
- [Explaining grokking through circuit efficiency](https://mlsanity.com/paper/2309.02390/) (2023)
- [Grokking as the Transition from Lazy to Rich Training Dynamics](https://mlsanity.com/paper/2310.06110/) (2023)
- [Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking](https://mlsanity.com/paper/2311.18817/) (2023)
- [Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization](https://mlsanity.com/paper/2405.15071/) (2024)
- [Grokfast: Accelerated Grokking by Amplifying Slow Gradients](https://mlsanity.com/paper/2405.20233/) (2024)
- [Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data](https://mlsanity.com/paper/2310.02541/) (2023)
- [Approaching Deep Learning through the Spectral Dynamics of Weights](https://mlsanity.com/paper/2408.11804/) (2024)
- [Emergence in non-neural models: grokking modular arithmetic via average gradient outer product](https://mlsanity.com/paper/2407.20199/) (2024)
- [Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding](https://mlsanity.com/paper/2310.16441/) (2023)
- [Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competition](https://mlsanity.com/paper/2402.15175/) (2024)
- [RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?](https://mlsanity.com/paper/2509.21016/) (2025)
- [Grokking Modular Polynomials](https://mlsanity.com/paper/2406.03495/) (2024)
- [On Catastrophic Inheritance of Large Foundation Models](https://mlsanity.com/paper/2402.01909/) (2024)
- [Grokking as Compression: A Nonlinear Complexity Perspective](https://mlsanity.com/paper/2310.05918/) (2023)
- [Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomials](https://mlsanity.com/paper/2402.16726/) (2024)
- [Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Model](https://mlsanity.com/paper/2504.13292/) (2025)
- [Understanding Grokking Through A Robustness Viewpoint](https://mlsanity.com/paper/2311.06597/) (2023)
- [Grokking at the Edge of Linear Separability](https://mlsanity.com/paper/2410.04489/) (2024)
- [Grokking in the Wild: Data Augmentation for Real-World Multi-Hop Reasoning with Transformers](https://mlsanity.com/paper/2504.20752/) (2025)
- [Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking](https://mlsanity.com/paper/2602.16746/) (2026)
- [The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structure](https://mlsanity.com/paper/2602.18523/) (2026)
- [What Can Grokking Teach Us About Learning Under Nonstationarity?](https://mlsanity.com/paper/2507.20057/) (2025)
- [To Grok Grokking: Provable Grokking in Ridge Regression](https://mlsanity.com/paper/2601.19791/) (2026)
- [Survival of the Fittest Representation: A Case Study with Modular Addition](https://mlsanity.com/paper/2405.17420/) (2024)
- [The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization](https://mlsanity.com/paper/2603.13331/) (2026)
- [Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networks](https://mlsanity.com/paper/2310.19470/) (2023)
- [Mechanistic Insights into Grokking from the Embedding Layer](https://mlsanity.com/paper/2505.15624/) (2025)
- [A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization](https://mlsanity.com/paper/2603.25009/) (2026)
- [How more data can hurt: Instability and regularization in next-generation reservoir computing](https://mlsanity.com/paper/2407.08641/) (2024)

## Abstract (arXiv)

A principled understanding of generalization in deep learning may require unifying disparate observations under a single conceptual framework. Previous work has studied grokking, a training dynamic in which a sustained period of near-perfect training performance and near-chance test performance is eventually followed by generalization, as well as the superficially similar double descent. These topics have so far been studied in isolation. We hypothesize that grokking and double descent can be understood as instances of the same learning dynamics within a framework of pattern learning speeds. We propose that this framework also applies when varying model capacity instead of optimization steps, and provide the first demonstration of model-wise grokking.