Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkin
Not on the map. Why?
Abstract (arXiv)
Grokking - the delayed transition from memorisation to generalisation in neural networks - remains poorly understood. We study this phenomenon through the geometry of learned representations and identify a consistent empirical signature preceding generalisation: collapse of the spectral entropy of the representation covariance matrix. Across modular arithmetic tasks and multiple random seeds, spectral entropy decreases gradually during training and crosses a stable task-specific threshold before test accuracy rises. A representation-mixing intervention that delays this collapse also delays grokking, including under norm-matched controls, indicating that the effect is not explained by parameter norm alone. We further show that the entropy gap predicts the remaining time until grokking with useful out-of-sample accuracy. To probe the structure underlying this transition, we introduce a Fourier-alignment observable for cyclic-group tasks. Entropy collapse is strongly coupled to the emergence of Fourier-aligned representations, suggesting that spectral entropy tracks concentration of the representation into task-structured directions rather than generic compression alone. The same qualitative dynamics appear in non-abelian group composition tasks, while MLP controls show that entropy collapse by itself is insufficient for grokking in the absence of appropriate inductive bias. Taken together, the results support a view of grokking as a representational phase transition with an observable geometric signature. We discuss the scope and limitations of this interpretation, connections to recent feature-learning and spectral-dynamics work, and directions for testing whether similar transitions appear in larger-scale learning systems.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 29 of its 41 references that are in the map. First posted on arXiv 13 Apr 2026.
Builds on
29 of 41 references.
References · 29
- 2017Attention Is All You Needcited by 60,361
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
- 2022In-context Learning and Induction Headscited by 964
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
- 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
- 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
- 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
- 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
- 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
- 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
- 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
17 more
- 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 146
- 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 144
- 2023Explaining grokking through circuit efficiencycited by 111
- 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 108
- 2023Grokking modular arithmeticcited by 83
- 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 83
- 2023Unifying Grokking and Double Descentcited by 50
- 2024Grokfast: Accelerated Grokking by Amplifying Slow Gradientscited by 41
- 2024Emergence in non-neural models: grokking modular arithmetic via average gradient outer productcited by 28
- 2024The Complexity Dynamics of Grokkingcited by 22
- 2025Grokking in LLM Pretraining? Monitor Memorization-to-Generalization without Testcited by 10
- 2025From SGD to Spectra: A Theory of Neural Network Weight Dynamicscited by 9
- 2025Grokking and Generalization Collapse: Insights from textttHTSR theorycited by 8
- 2024Stochastic weight matrix dynamics during learning and Dyson Brownian motioncited by 7
- 2025Entropy Collapse: A Universal Failure Mode of Intelligent Systemscited by 2
- 2026Norm-Hierarchy Transitions in Representation Learning: When and Why Neural Networks Abandon Shortcutscited by 2
- 2026Breaking Data Symmetry is Needed For Generalization in Feature Learning Kernelscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.