mlsanityData to 7 Oct 2026

Unifying Grokking and Double Descent

arXiv 2303.06173 ↗First posted 10 Mar 2023cs.LG cs.AIcited by 49

Not on the map. Why?

Abstract (arXiv)

A principled understanding of generalization in deep learning may require unifying disparate observations under a single conceptual framework. Previous work has studied grokking, a training dynamic in which a sustained period of near-perfect training performance and near-chance test performance is eventually followed by generalization, as well as the superficially similar double descent. These topics have so far been studied in isolation. We hypothesize that grokking and double descent can be understood as instances of the same learning dynamics within a framework of pattern learning speeds. We propose that this framework also applies when varying model capacity instead of optimization steps, and provide the first demonstration of model-wise grokking.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 14 of its 25 references that are in the map. First posted on arXiv 10 Mar 2023.

Builds on

14 of 25 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2017A Closer Look at Memorization in Deep Networkscited by 936
  4. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  5. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  6. 2021Explaining Neural Scaling Lawscited by 353
  7. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  8. 2019Optimal Policies Tend to Seek Powercited by 115
  9. 2022Is Power-Seeking AI an Existential Risk?cited by 101
  10. 2022A Solvable Model of Neural Scaling Lawscited by 101
  11. 2020Early Stopping in Deep Networks: Double Descent and How to Eliminate itcited by 40
  12. 2022Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamicscited by 26
2 more
  1. 2021Multi-scale Feature Learning Dynamics: Insights for Double Descentcited by 24
  2. 2021When and how epochwise double descent happenscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 13; 2025: 12; 2026: 15.

49 papers.

Sort:

By area:

  1. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  2. 2023Explaining grokking through circuit efficiencycited by 105
  3. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  4. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  5. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  6. 2024Grokfast: Accelerated Grokking by Amplifying Slow Gradientscited by 38
  7. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 32
  8. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  9. 2024Emergence in non-neural models: grokking modular arithmetic via average gradient outer productcited by 26
  10. 2023Grokking in Linear Estimators -- A Solvable Model that Groks without Understandingcited by 25
  11. 2024Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competitioncited by 23
  12. 2025RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?cited by 23
37 more
  1. 2024Grokking Modular Polynomialscited by 18
  2. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  3. 2023Grokking as Compression: A Nonlinear Complexity Perspectivecited by 14
  4. 2024Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomialscited by 14
  5. 2025Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Modelcited by 13
  6. 2025Grokking in the Wild: Data Augmentation for Real-World Multi-Hop Reasoning with Transformerscited by 12
  7. 2024Grokking at the Edge of Linear Separabilitycited by 11
  8. 2023Understanding Grokking Through A Robustness Viewpointcited by 10
  9. 2026Low-Dimensional and Transversely Curved Optimization Dynamics in Grokkingcited by 10
  10. 2026The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structurecited by 8
  11. 2024Survival of the Fittest Representation: A Case Study with Modular Additioncited by 7
  12. 2025What Can Grokking Teach Us About Learning Under Nonstationarity?cited by 7
  13. 2025Mechanistic Insights into Grokking from the Embedding Layercited by 6
  14. 2026To Grok Grokking: Provable Grokking in Ridge Regressioncited by 6
  15. 2026The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalizationcited by 6
  16. 2023Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networkscited by 5
  17. 2024How more data can hurt: Instability and regularization in next-generation reservoir computingcited by 5
  18. 2026A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularizationcited by 5
  19. 2023The semantic landscape paradigm for neural networkscited by 4
  20. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
  21. 2026Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speedscited by 4
  22. 2025NeuralGrok: Accelerate Grokking by Neural Gradient Transformationcited by 2
  23. 2026Explaining Grokking in Transformers through the Lens of Inductive Biascited by 2
  24. 2026Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkincited by 2
  25. 2026Circuit Synchronization Precedes Generalization: A Causal Precursor to Grokkingcited by 2
  26. 2025A Spin Glass Characterization of Neural Networkscited by 1
  27. 2025Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokkingcited by 1
  28. 2026Unveiling Memorization-Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noisecited by 1
  29. 2024One system for learning and remembering episodes and rulescited by 0
  30. 2025Decomposing Prediction Mechanisms for In-Context Recallcited by 0
  31. 2025Foundations of Artificial Intelligence Frameworks: Notion and Limits of AGIcited by 0
  32. 2025Grokked Models are Better Unlearnerscited by 0
  33. 2026Deep Exploration of Epoch-wise Double Descent in Noisy Data: Signal Separation, Large Activation, and Benign Overfittingcited by 0
  34. 2026Global Low-Rank, Local Full-Rank: The Holographic Encoding of Learned Algorithmscited by 0
  35. 2026Grokking of Diffusion Models: Case Study on Modular Additioncited by 0
  36. 2026A Training-Time Diagnostic for Generalization via the Log-Alignment Ratiocited by 0
  37. 2026Is Grokking a Loss of Normal Hyperbolicity of the Interpolation Manifold?cited by 0