mlsanityData to 14 Oct 2026

Omnigrok: Grokking Beyond Algorithmic Data

arXiv 2210.01117 ↗First posted 3 Oct 2022cs.LG cs.AI physics.data-ancited by 146

Not on the map. Why?

Abstract (arXiv)

Grokking, the unusual phenomenon for algorithmic datasets where generalization happens long after overfitting the training data, has remained elusive. We aim to understand grokking by analyzing the loss landscapes of neural networks, identifying the mismatch between training and test losses as the cause for grokking. We refer to this as the "LU mechanism" because training and test losses (against model weight norm) typically resemble "L" and "U", respectively. This simple mechanism can nicely explain many aspects of grokking: data size dependence, weight decay dependence, the emergence of representations, etc. Guided by the intuitive picture, we are able to induce grokking on tasks involving images, language and molecules. In the reverse direction, we are able to eliminate grokking for algorithmic datasets. We attribute the dramatic nature of grokking for algorithmic datasets to representation learning.

Key facts. Cited by 146 papers in the map (data through 14 Oct 2026). Builds on 20 of its 32 references that are in the map. First posted on arXiv 3 Oct 2022.

Builds on

20 of 32 references.

References · 20

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  4. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  5. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  6. 2016Deep Information Propagationcited by 195
  7. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  8. 2018Training behavior of deep neural network in frequency domaincited by 174
  9. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  10. 2018Three Mechanisms of Weight Decay Regularizationcited by 113
  11. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 94
  12. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
8 more
  1. 2016Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNscited by 65
  2. 2020Triple descent and the two kinds of overfitting: Where & why do they appear?cited by 65
  3. 2020Multiple Descent: Design Your Own Generalization Curvecited by 45
  4. 2020On the training dynamics of deep networks with L2 regularizationcited by 39
  5. 2019A type of generalization error induced by initialization in deep neural networkscited by 33
  6. 2018The Goldilocks zone: Towards better understanding of neural network loss landscapescited by 32
  7. 2022Regularization-wise double descent: Why it occurs and how to eliminate itcited by 9
  8. 2020Revisiting Initialization of Neural Networkscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 27; 2024: 29; 2025: 31; 2026: 59.

146 papers.

Sort:

By area:

  1. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  3. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 156
  4. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 124
  5. 2023Explaining grokking through circuit efficiencycited by 111
  6. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 108
  7. 2023Grokking modular arithmeticcited by 83
  8. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 83
  9. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 80
  10. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 79
  11. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 56
  12. 2023Grokking of Hierarchical Structure in Vanilla Transformerscited by 53
38 more
  1. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  2. 2025Grokking at the Edge of Numerical Stabilitycited by 46
  3. 2024Grokfast: Accelerated Grokking by Amplifying Slow Gradientscited by 41
  4. 2023Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretabilitycited by 38
  5. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 36
  6. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 35
  7. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 33
  8. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  9. 2024Emergence in non-neural models: grokking modular arithmetic via average gradient outer productcited by 28
  10. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  11. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  12. 2025Towards Understanding Distilled Reasoning Models: A Representational Approachcited by 26
  13. 2023To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasetscited by 24
  14. 2025RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?cited by 23
  15. 2025Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesiscited by 22
  16. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 20
  17. 2024Grokking Modular Polynomialscited by 18
  18. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 17
  19. 2023Latent State Models of Training Dynamicscited by 16
  20. 2025Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokkingcited by 15
  21. 2023Grokking as Compression: A Nonlinear Complexity Perspectivecited by 14
  22. 2023Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problemscited by 14
  23. 2024Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomialscited by 14
  24. 2025Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetriescited by 14
  25. 2023Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luckcited by 13
  26. 2023Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexitycited by 13
  27. 2025Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Modelcited by 13
  28. 2025A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisationcited by 13
  29. 2023Understanding Grokking Through A Robustness Viewpointcited by 12
  30. 2024Emergent properties with repeated examplescited by 11
  31. 2024Abrupt Learning in Transformers: A Case Study on Matrix Completioncited by 11
  32. 2026Low-Dimensional and Transversely Curved Optimization Dynamics in Grokkingcited by 11
  33. 2025Grokking in LLM Pretraining? Monitor Memorization-to-Generalization without Testcited by 10
  34. 2023Identifying Equivalent Training Dynamicscited by 9
  35. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
  36. 2025The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifoldcited by 9
  37. 2026The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structurecited by 9
  38. 2024Progress Measures for Grokking on Real-world Taskscited by 8

50 most cited of 146 All 146 (JSON)