mlsanityData to 7 Oct 2026

Grokking as a First Order Phase Transition in Two Layer Networks

arXiv 2310.03789 ↗First posted 5 Oct 2023stat.ML cond-mat.dis-nn cs.LGcited by 48

Not on the map. Why?

Abstract (arXiv)

A key property of deep neural networks (DNNs) is their ability to learn new features during training. This intriguing aspect of deep learning stands out most clearly in recently reported Grokking phenomena. While mainly reflected as a sudden increase in test accuracy, Grokking is also believed to be a beyond lazy-learning/Gaussian Process (GP) phenomenon involving feature learning. Here we apply a recent development in the theory of feature learning, the adaptive kernel approach, to two teacher-student models with cubic-polynomial and modular addition teachers. We provide analytical predictions on feature learning and Grokking properties of these models and demonstrate a mapping between Grokking and the theory of phase transitions. We show that after Grokking, the state of the DNN is analogous to the mixed phase following a first-order phase transition. In this mixed phase, the DNN generates useful internal representations of the teacher that are sharply distinct from those before the transition.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 24 of its 38 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

24 of 38 references.

References · 24

  1. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  2. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  3. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  4. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  6. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  7. 2015Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithmcited by 195
  8. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  9. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  10. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  11. 2023Explaining grokking through circuit efficiencycited by 105
  12. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
12 more
  1. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  2. 2023Grokking modular arithmeticcited by 77
  3. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  4. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  5. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  6. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  7. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  8. 2019Online Stochastic Gradient Descent with Arbitrary Initialization Solves Non-smooth, Non-convex Phase Retrievalcited by 34
  9. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  10. 2020Predicting the outputs of finite deep neural networks trained with noisy gradientscited by 31
  11. 2022Grokking phase transitions in learning local rules with gradient descentcited by 28
  12. 2019Stationary Points of Shallow Neural Networks with Quadratic Activation Functioncited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 21; 2026: 16.

48 papers.

Sort:

  1. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  2. 2025Grokking at the Edge of Numerical Stabilitycited by 41
  3. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 34
  4. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  5. 2024Grokking Modular Polynomialscited by 18
  6. 2024Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomialscited by 14
  7. 2025Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokkingcited by 14
  8. 2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
  9. 2024Coding schemes in neural networks learning classification taskscited by 11
  10. 2024Grokking at the Edge of Linear Separabilitycited by 11
  11. 2025From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learningcited by 10
  12. 2025Statistical mechanics of extensive-width Bayesian neural networks near interpolationcited by 9
36 more
  1. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 8
  2. 2025Is Grokking a Computational Glass Relaxation?cited by 8
  3. 2024Phase Transitions in the Output Distribution of Large Language Modelscited by 7
  4. 2023Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networkscited by 5
  5. 2026Noise-Driven Escape from Metastable Phases explains Grokking in Deep Neural Networkscited by 5
  6. 2024Can Kernel Methods Explain How the Data Affects Neural Collapse?cited by 4
  7. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
  8. 2025Mitigating the Curse of Detail: Scaling Arguments for Feature Learning and Sample Complexitycited by 4
  9. 2026A Basin-Selection Perspective on Grokking via Singular Learning Theorycited by 4
  10. 2026Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speedscited by 4
  11. 2025Learning words in groups: fusion algebras, tensor ranks and grokkingcited by 3
  12. 2026Towards Worst-Case Guarantees with Scale-Aware Interpretabilitycited by 3
  13. 2026Grokking as a Falsifiable Finite-Size Transitioncited by 3
  14. 2024Over the Edge of Chaos? Excess Complexity as a Roadblock to Artificial General Intelligencecited by 2
  15. 2024Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurementcited by 2
  16. 2025Grokking vs. Learning: Same Features, Different Encodingscited by 2
  17. 2025GrokAlign: Geometric Characterisation and Acceleration of Grokkingcited by 2
  18. 2025Transfer Learning in Infinite Width Feature Learning Networkscited by 2
  19. 2026Statistical Properties of Training & Generalizationcited by 2
  20. 2026Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activationscited by 2
  21. 2025Rethinking Over-Smoothing in Graph Neural Networks: A Perspective from Anderson Localizationcited by 1
  22. 2025When Data Falls Short: Grokking Below the Critical Thresholdcited by 1
  23. 2026What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropycited by 1
  24. 2026From Symmetry to Invariance: Learning Galois Equivalent Representations in Finite Fieldscited by 1
  25. 2025Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolationcited by 0
  26. 2025Phase Transitions between Accuracy Regimes in L2 regularized Deep Neural Networkscited by 0
  27. 2025Consciousness as a Jamming Phasecited by 0
  28. 2025A simple mean field model of feature learningcited by 0
  29. 2025Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolationcited by 0
  30. 2025Evidence of Phase Transitions in Small Transformer-Based Language Modelscited by 0
  31. 2026Generalization at the Edge of Stabilitycited by 0
  32. 2026The Geometric Structure of Models Learning Sparse Datacited by 0
  33. 2026The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Modelscited by 0
  34. 2026A Function-Space Approach to the Statistical Mechanics of Learning Dynamicscited by 0
  35. 2026An Exploratory Replica-Overlap Probe of the Grokking Transitioncited by 0
  36. 2026A Spectral Theory of Grokking: Weight Decay induces Feature Learningcited by 0