mlsanityData to 7 Oct 2026

Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition

arXiv 2408.08944 ↗First posted 16 Aug 2024cs.LG cs.IT math.ITcited by 16

Not on the map. Why?

Abstract (arXiv)

This paper studies emergent phenomena in neural networks by focusing on grokking where models suddenly generalize after delayed memorization. To understand this phase transition, we utilize higher-order mutual information to analyze the collective behavior (synergy) and shared properties (redundancy) between neurons during training. We identify distinct phases before grokking allowing us to anticipate when it occurs. We attribute grokking to an emergent phase transition caused by the synergistic interactions between neurons as a whole. We show that weight decay and weight initialization can enhance the emergent phase.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 27 of its 32 references that are in the map. First posted on arXiv 16 Aug 2024.

Builds on

27 of 32 references.

References · 27

  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2017Decoupled Weight Decay Regularizationcited by 1,722
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  5. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  6. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  7. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  8. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  9. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  10. 2020Optimal Regularization Can Mitigate Double Descentcited by 107
  11. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  12. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 78
15 more
  1. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  2. 2018The Goldilocks zone: Towards better understanding of neural network loss landscapescited by 32
  3. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 32
  4. 2023Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grokcited by 31
  5. 2019Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Informationcited by 30
  6. 2023To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasetscited by 24
  7. 2024Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competitioncited by 23
  8. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  9. 2023Grokking as Compression: A Nonlinear Complexity Perspectivecited by 14
  10. 2023Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexitycited by 13
  11. 2022Clustering units in neural networks: upstream vs downstream informationcited by 11
  12. 2021Quantifying Local Specialization in Deep Neural Networkscited by 10
  13. 2023Understanding Grokking Through A Robustness Viewpointcited by 10
  14. 2024Deconstructing the Goldilocks Zone of Neural Network Initializationcited by 4
  15. 2022Higher-order mutual information reveals synergistic sub-networks for multi-neuron importancecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 11.

16 papers.

Sort:

  1. 2025Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokkingcited by 14
  2. 2026Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speedscited by 4
  3. 2025TRACE for Tracking the Emergence of Semantic Representations in Transformerscited by 3
  4. 2026Grokking as a Falsifiable Finite-Size Transitioncited by 3
  5. 2026Topological Signatures of Grokkingcited by 2
  6. 2025Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokkingcited by 1
  7. 2025A Review of Developmental Interpretability in Large Language Modelscited by 1
  8. 2026Grokking as Structural Inference: Transformers Need Bayesian Lottery Ticketscited by 1
  9. 2026Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reductioncited by 1
  10. 2026What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropycited by 1
  11. 2024Do Mice Grok? Glimpses of Hidden Progress During Overtraining in Sensory Cortexcited by 0
  12. 2026A Bayesian Perspective on the Role of Epistemic Uncertainty for Delayed Generalization in In-Context Learningcited by 0
4 more
  1. 2026Phase Transitions in Driven Informational Systems: A Two-Field Perspective on Learning Theory and Non-Equilibrium Chemistrycited by 0
  2. 2026Emergence via Phase Transitions: Mechanism Landscapes and Universal Convergence Across Complex Systemscited by 0
  3. 2026Scalar Representations of Neural Network Training Dynamicscited by 0
  4. 2026Emergent Generalization by Representation Learning in Artificial Neural Networkscited by 0