mlsanityData to 14 Oct 2026

Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking

arXiv 2311.18817 ↗First posted 30 Nov 2023cs.LG cs.AIcited by 80

Not on the map. Why?

Abstract (arXiv)

Recent work by Power et al. (2022) highlighted a surprising "grokking" phenomenon in learning arithmetic tasks: a neural net first "memorizes" the training set, resulting in perfect training accuracy but near-random test accuracy, and after training for sufficiently longer, it suddenly transitions to perfect test accuracy. This paper studies the grokking phenomenon in theoretical setups and shows that it can be induced by a dichotomy of early and late phase implicit biases. Specifically, when training homogeneous neural nets with large initialization and small weight decay on both classification and regression tasks, we prove that the training process gets trapped at a solution corresponding to a kernel predictor for a long time, and then a very sharp transition to min-norm/max-margin predictors occurs, leading to a dramatic change in test accuracy.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Builds on 58 of its 73 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

58 of 73 references.

References · 58

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  5. 2018On Lazy Training in Differentiable Programmingcited by 675
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  7. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  8. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  10. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  11. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  12. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
38 more
  1. 2017Implicit Regularization in Matrix Factorizationcited by 322
  2. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  3. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  4. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  5. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  6. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  7. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  8. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  9. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  10. 2020Directional convergence and alignment in deep learningcited by 178
  11. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  12. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 146
  13. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 144
  14. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  15. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 121
  16. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  17. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 119
  18. 2023Explaining grokking through circuit efficiencycited by 111
  19. 2018Convergence of Gradient Descent on Separable Datacited by 110
  20. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 108
  21. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  22. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 94
  23. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  24. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  25. 2023Grokking modular arithmeticcited by 83
  26. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 83
  27. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 71
  28. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 70
  29. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 65
  30. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  31. 2023Unifying Grokking and Double Descentcited by 50
  32. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 46
  33. 2020On the training dynamics of deep networks with L2 regularizationcited by 39
  34. 2023Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grokcited by 35
  35. 2023Learning the greatest common divisor: explaining transformer predictionscited by 33
  36. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 33
  37. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  38. 2018The Goldilocks zone: Towards better understanding of neural network loss landscapescited by 32

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 22; 2025: 22; 2026: 36.

80 papers.

Sort:

By area:

  1. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 118
  2. 2025Grokking at the Edge of Numerical Stabilitycited by 46
  3. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  4. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 36
  5. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 31
  6. 2024Emergence in non-neural models: grokking modular arithmetic via average gradient outer productcited by 28
  7. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  8. 2024Towards Uncovering How Large Language Model Works: An Explainability Perspectivecited by 22
  9. 2024Grokking Modular Polynomialscited by 18
  10. 2024Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transitioncited by 17
  11. 2024Deep Grokking: Would Deep Neural Networks Generalize Better?cited by 15
  12. 2024Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomialscited by 14
38 more
  1. 2025Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Modelcited by 13
  2. 2025A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisationcited by 13
  3. 2024Grokking at the Edge of Linear Separabilitycited by 12
  4. 2025Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?cited by 12
  5. 2024Simplicity Bias of Two-Layer Networks beyond Linearly Separable Datacited by 11
  6. 2026Low-Dimensional and Transversely Curved Optimization Dynamics in Grokkingcited by 11
  7. 2024Feature learning as alignment: a structural property of gradient descent in non-linear neural networkscited by 10
  8. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
  9. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  10. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 9
  11. 2026The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structurecited by 9
  12. 2025What Can Grokking Teach Us About Learning Under Nonstationarity?cited by 8
  13. 2025Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokkingcited by 8
  14. 2026To Grok Grokking: Provable Grokking in Ridge Regressioncited by 8
  15. 2025What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformerscited by 7
  16. 2026Early-Warning Signals of Grokking via Loss-Landscape Geometrycited by 7
  17. 2026The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalizationcited by 7
  18. 2025Mechanistic Insights into Grokking from the Embedding Layercited by 6
  19. 2025On Learning Verifiers and Implications to Chain-of-Thought Reasoningcited by 6
  20. 2026Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scalescited by 6
  21. 2026Noise-Driven Escape from Metastable Phases explains Grokking in Deep Neural Networkscited by 5
  22. 2025Learning words in groups: fusion algebras, tensor ranks and grokkingcited by 4
  23. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
  24. 2026A Basin-Selection Perspective on Grokking via Singular Learning Theorycited by 4
  25. 2026Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikescited by 4
  26. 2026Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speedscited by 4
  27. 2026The Weight Norm Sets the Grokking Timescale: A Causal Delay Lawcited by 4
  28. 2024Acceleration of Grokking in Learning Arithmetic Operations via Kolmogorov-Arnold Representationcited by 3
  29. 2024A rationale from frequency perspective for grokking in training neural networkcited by 3
  30. 2025Not All Explanations for Deep Learning Phenomena Are Equally Valuablecited by 3
  31. 2026Grokking as a Falsifiable Finite-Size Transitioncited by 3
  32. 2026Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learningcited by 3
  33. 2024Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurementcited by 2
  34. 2025GrokAlign: Geometric Characterisation and Acceleration of Grokkingcited by 2
  35. 2025Precise Dynamics of Diagonal Linear Networks: A Unifying Analysis by Dynamical Mean-Field Theorycited by 2
  36. 2026Grokking Finite-Dimensional Algebracited by 2
  37. 2026Norm-Hierarchy Transitions in Representation Learning: When and Why Neural Networks Abandon Shortcutscited by 2
  38. 2026Distributional Spectral Diagnostics for Localizing Grokking Transitionscited by 2

50 most cited of 80 All 80 (JSON)