# Grokking and Generalization Collapse: Insights from textttHTSR theory

arXiv: https://arxiv.org/abs/2506.04434 · HTML: https://mlsanity.com/paper/2506.04434/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 13 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Builds on (most cited first, 7 of 7)

- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers](https://mlsanity.com/paper/2210.06313/) (2022)
- [Explaining grokking through circuit efficiency](https://mlsanity.com/paper/2309.02390/) (2023)
- [A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworks](https://mlsanity.com/paper/2303.11873/) (2023)
- [Progress Measures for Grokking on Real-world Tasks](https://mlsanity.com/paper/2405.12755/) (2024)
- [A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustness](https://mlsanity.com/paper/2309.03004/) (2023)
- [The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks](https://mlsanity.com/paper/2104.06153/) (2021)

## Built on by (most cited first, 8 of 8)

- [Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes](https://mlsanity.com/paper/2605.06152/) (2026)
- [Late-Stage Generalization Collapse in Grokking: Detecting anti-grokking with Weightwatcher](https://mlsanity.com/paper/2602.02859/) (2026)
- [Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkin](https://mlsanity.com/paper/2604.13123/) (2026)
- [Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers](https://mlsanity.com/paper/2608.07436/) (2026)
- [Early-stopping for Transformer model training](https://mlsanity.com/paper/2510.16074/) (2025)
- [Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theory](https://mlsanity.com/paper/2512.12911/) (2025)
- [Fast Generalization after Interpolation via Critically Damped Momentum Optimization](https://mlsanity.com/paper/2606.01521/) (2026)
- [Predicting Privacy Leakage from Weight Spectral Density](https://mlsanity.com/paper/2609.11780/) (2026)

## Abstract (arXiv)

We study the well-known grokking phenomena in neural networks (NNs) using a 3-layer MLP trained on 1 k-sample subset of MNIST, with and without weight decay, and discover a novel third phase -- anti-grokking -- that occurs very late in training and resembles but is distinct from the familiar pre-grokking phases: test accuracy collapses while training accuracy stays perfect. This late-stage collapse is distinct, from the known pre-grokking and grokking phases, and is not detected by other proposed grokking progress measures. Leveraging Heavy-Tailed Self-Regularization HTSR through the open-source WeightWatcher tool, we show that the HTSR layer quality metric α alone delineates all three phases, whereas the best competing metrics detect only the first two. The anti-grokking is revealed by training for 107 and is invariably heralded by α < 2 and the appearance of Correlation Traps -- outlier singular values in the randomized layer weight matrices that make the layer weight matrix atypical and signal overfitting of the training set. Such traps are verified by visual inspection of the layer-wise empirical spectral densities, and by using Kolmogorov--Smirnov tests on randomized spectra. Comparative metrics, including activation sparsity, absolute weight entropy, circuit complexity, and l2 weight norms track pre-grokking and grokking but fail to distinguish grokking from anti-grokking. This discovery provides a way to measure overfitting and generalization collapse without direct access to the test data. These results strengthen the claim that the HTSR α provides universal layer-convergence target at α ≈ 2 and underscore the value of using the HTSR alpha (α) metric as a measure of generalization.