# Grokking phase transitions in learning local rules with gradient descent

arXiv: https://arxiv.org/abs/2210.15435 · HTML: https://mlsanity.com/paper/2210.15435/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 51 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 30 of 41)

- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training](https://mlsanity.com/paper/2101.12699/) (2021)
- [Neural collapse with unconstrained features](https://mlsanity.com/paper/2011.11619/) (2020)
- [Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models](https://mlsanity.com/paper/1708.03395/) (2017)
- [On the Expressive Power of Deep Learning: A Tensor Analysis](https://mlsanity.com/paper/1509.05009/) (2015)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon](https://mlsanity.com/paper/2206.04817/) (2022)
- [Triple descent and the two kinds of overfitting: Where & why do they appear?](https://mlsanity.com/paper/2006.03509/) (2020)
- [Tree Tensor Networks for Generative Modeling](https://mlsanity.com/paper/1901.02217/) (2019)
- [Limitations of Neural Collapse for Understanding Generalization in Deep Learning](https://mlsanity.com/paper/2202.08384/) (2022)
- [Expressive power of tensor-network factorizations for probabilistic modeling, with applications from hidden Markov models to quantum machine learning](https://mlsanity.com/paper/1907.03741/) (2019)
- [Learning Relevant Features of Data with Multi-scale Tensor Networks](https://mlsanity.com/paper/1801.00315/) (2017)
- [Machine Learning by Unitary Tensor Network of Hierarchical Tree Structure](https://mlsanity.com/paper/1710.04833/) (2017)
- [Supervised Learning with Quantum-Inspired Tensor Networks](https://mlsanity.com/paper/1605.05775/) (2016)
- [TensorNetwork for Machine Learning](https://mlsanity.com/paper/1906.06329/) (2019)
- [Equivalence of restricted Boltzmann machines and tensor network states](https://mlsanity.com/paper/1701.04831/) (2017)
- [Anomaly Detection with Tensor Networks](https://mlsanity.com/paper/2006.02516/) (2020)
- [Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Design](https://mlsanity.com/paper/1704.01552/) (2017)
- [Multi-scale Feature Learning Dynamics: Insights for Double Descent](https://mlsanity.com/paper/2112.03215/) (2021)
- [Generative Tensor Network Classification Model for Supervised Machine Learning](https://mlsanity.com/paper/1903.10742/) (2019)
- [Tensor networks for unsupervised machine learning](https://mlsanity.com/paper/2106.12974/) (2021)
- [Modeling Sequences with Quantum States: A Look Under the Hood](https://mlsanity.com/paper/1910.07425/) (2019)
- [Entanglement and Tensor Networks for Supervised Image Classification](https://mlsanity.com/paper/2007.06082/) (2020)

## Built on by (most cited first, 28 of 28)

- [Grokking modular arithmetic](https://mlsanity.com/paper/2301.02679/) (2023)
- [Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking](https://mlsanity.com/paper/2311.18817/) (2023)
- [Grokking as a First Order Phase Transition in Two Layer Networks](https://mlsanity.com/paper/2310.03789/) (2023)
- [Grokking at the Edge of Numerical Stability](https://mlsanity.com/paper/2501.04697/) (2025)
- [Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok](https://mlsanity.com/paper/2306.13253/) (2023)
- [Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data](https://mlsanity.com/paper/2310.02541/) (2023)
- [To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets](https://mlsanity.com/paper/2310.13061/) (2023)
- [Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomials](https://mlsanity.com/paper/2402.16726/) (2024)
- [Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexity](https://mlsanity.com/paper/2310.17247/) (2023)
- [Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Model](https://mlsanity.com/paper/2504.13292/) (2025)
- [Understanding Grokking Through A Robustness Viewpoint](https://mlsanity.com/paper/2311.06597/) (2023)
- [Grokking at the Edge of Linear Separability](https://mlsanity.com/paper/2410.04489/) (2024)
- [Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking](https://mlsanity.com/paper/2509.17738/) (2025)
- [To Grok Grokking: Provable Grokking in Ridge Regression](https://mlsanity.com/paper/2601.19791/) (2026)
- [Understanding the Generalization Benefits of Late Learning Rate Decay](https://mlsanity.com/paper/2401.11600/) (2024)
- [A Matrix Product State Model for Simultaneous Classification and Generation](https://mlsanity.com/paper/2406.17441/) (2024)
- [The semantic landscape paradigm for neural networks](https://mlsanity.com/paper/2307.09550/) (2023)
- [A Basin-Selection Perspective on Grokking via Singular Learning Theory](https://mlsanity.com/paper/2603.01192/) (2026)
- [Positive unlabeled learning with tensor networks](https://mlsanity.com/paper/2211.14085/) (2022)
- [Grokking as a Falsifiable Finite-Size Transition](https://mlsanity.com/paper/2603.24746/) (2026)
- [Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics](https://mlsanity.com/paper/2605.20441/) (2026)
- [Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations](https://mlsanity.com/paper/2607.13749/) (2026)
- [Grokking Explained: A Statistical Phenomenon](https://mlsanity.com/paper/2502.01774/) (2025)
- [Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction](https://mlsanity.com/paper/2606.05863/) (2026)
- [What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy](https://mlsanity.com/paper/2606.18465/) (2026)
- [BlueGlass: A Framework for Composite AI Safety](https://mlsanity.com/paper/2507.10106/) (2025)
- [Mean-Field Dynamics of Chain-of-Thought Reasoning in Large Language Models](https://mlsanity.com/paper/2608.05152/) (2026)
- [An Exploratory Replica-Overlap Probe of the Grokking Transition](https://mlsanity.com/paper/2609.25634/) (2026)

## Abstract (arXiv)

We discuss two solvable grokking (generalisation beyond overfitting) models in a rule learning scenario. We show that grokking is a phase transition and find exact analytic expressions for the critical exponents, grokking probability, and grokking time distribution. Further, we introduce a tensor-network map that connects the proposed grokking setup with the standard (perceptron) statistical learning theory and show that grokking is a consequence of the locality of the teacher model. As an example, we analyse the cellular automata learning task, numerically determine the critical exponent and the grokking time distributions and compare them with the prediction of the proposed grokking model. Finally, we numerically analyse the connection between structure formation and grokking.