mlsanityData to 7 Oct 2026

Grokking phase transitions in learning local rules with gradient descent

arXiv 2210.15435 ↗First posted 26 Oct 2022cond-mat.stat-mech cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

We discuss two solvable grokking (generalisation beyond overfitting) models in a rule learning scenario. We show that grokking is a phase transition and find exact analytic expressions for the critical exponents, grokking probability, and grokking time distribution. Further, we introduce a tensor-network map that connects the proposed grokking setup with the standard (perceptron) statistical learning theory and show that grokking is a consequence of the locality of the teacher model. As an example, we analyse the cellular automata learning task, numerically determine the critical exponent and the grokking time distributions and compare them with the prediction of the proposed grokking model. Finally, we numerically analyse the connection between structure formation and grokking.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 41 of its 51 references that are in the map. First posted on arXiv 26 Oct 2022.

Builds on

41 of 51 references.

References · 41

  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  2. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  3. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  5. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  6. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  7. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  8. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  9. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  10. 2020Neural collapse with unconstrained featurescited by 140
  11. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 115
  12. 2015On the Expressive Power of Deep Learning: A Tensor Analysiscited by 97
29 more
  1. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  2. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  3. 2020Triple descent and the two kinds of overfitting: Where & why do they appear?cited by 64
  4. 2019Tree Tensor Networks for Generative Modelingcited by 63
  5. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  6. 2019Expressive power of tensor-network factorizations for probabilistic modeling, with applications from hidden Markov models to quantum machine learningcited by 55
  7. 2017Learning Relevant Features of Data with Multi-scale Tensor Networkscited by 50
  8. 2017Machine Learning by Unitary Tensor Network of Hierarchical Tree Structurecited by 39
  9. 2016Supervised Learning with Quantum-Inspired Tensor Networkscited by 35
  10. 2019TensorNetwork for Machine Learningcited by 29
  11. 2017Equivalence of restricted Boltzmann machines and tensor network statescited by 26
  12. 2020Anomaly Detection with Tensor Networkscited by 26
  13. 2021Multi-scale Feature Learning Dynamics: Insights for Double Descentcited by 24
  14. 2017Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Designcited by 22
  15. 2019Generative Tensor Network Classification Model for Supervised Machine Learningcited by 22
  16. 2021Tensor networks for unsupervised machine learningcited by 20
  17. 2019Modeling Sequences with Quantum States: A Look Under the Hoodcited by 19
  18. 2020Entanglement and Tensor Networks for Supervised Image Classificationcited by 18
  19. 2022Extracting Finite Automata from RNNs Using State Mergingcited by 17
  20. 2022From Tensor Network Quantum States to Tensorial Recurrent Neural Networkscited by 13
  21. 2019Probabilistic Modeling with Matrix Product Statescited by 11
  22. 2017Tensor network language modelcited by 9
  23. 2020Quantum Tensor Networks, Stochastic Processes, and Weighted Automatacited by 9
  24. 2020Residual Matrix Product State for Machine Learningcited by 9
  25. 2021Quantum Tensor Network in Machine Learning: An Application to Tiny Object Classificationcited by 9
  26. 2020Language Modeling with Reduced Densitiescited by 5
  27. 2021Tensor network to learn the wavefunction of datacited by 4
  28. 2020Network Anomaly Detection based on Tensor Decompositioncited by 3
  29. 2022Deep tensor networks with matrix product operatorscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 9; 2024: 4; 2025: 5; 2026: 9.

28 papers.

Sort:

By area:

  1. 2023Grokking modular arithmeticcited by 77
  2. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  3. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  4. 2025Grokking at the Edge of Numerical Stabilitycited by 41
  5. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 32
  6. 2023Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grokcited by 31
  7. 2023To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasetscited by 24
  8. 2024Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomialscited by 14
  9. 2023Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexitycited by 13
  10. 2025Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Modelcited by 13
  11. 2024Grokking at the Edge of Linear Separabilitycited by 11
  12. 2023Understanding Grokking Through A Robustness Viewpointcited by 10
16 more
  1. 2025Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokkingcited by 8
  2. 2024Understanding the Generalization Benefits of Late Learning Rate Decaycited by 6
  3. 2026To Grok Grokking: Provable Grokking in Ridge Regressioncited by 6
  4. 2024A Matrix Product State Model for Simultaneous Classification and Generationcited by 5
  5. 2023The semantic landscape paradigm for neural networkscited by 4
  6. 2026A Basin-Selection Perspective on Grokking via Singular Learning Theorycited by 4
  7. 2022Positive unlabeled learning with tensor networkscited by 3
  8. 2026Grokking as a Falsifiable Finite-Size Transitioncited by 3
  9. 2026Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnosticscited by 2
  10. 2026Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activationscited by 2
  11. 2025Grokking Explained: A Statistical Phenomenoncited by 1
  12. 2026Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reductioncited by 1
  13. 2026What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropycited by 1
  14. 2025BlueGlass: A Framework for Composite AI Safetycited by 0
  15. 2026Mean-Field Dynamics of Chain-of-Thought Reasoning in Large Language Modelscited by 0
  16. 2026An Exploratory Replica-Overlap Probe of the Grokking Transitioncited by 0