# Grokking From Abstraction to Intelligence

arXiv: https://arxiv.org/abs/2603.29262 · HTML: https://mlsanity.com/paper/2603.29262/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 31 Mar 2026.

## Builds on (most cited first, 23 of 23)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit](https://mlsanity.com/paper/2207.08799/) (2022)
- [The Quantization Model of Neural Scaling](https://mlsanity.com/paper/2303.13506/) (2023)
- [Explaining grokking through circuit efficiency](https://mlsanity.com/paper/2309.02390/) (2023)
- [Deep Learning is Singular, and That's Good](https://mlsanity.com/paper/2010.11560/) (2020)
- [Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models](https://mlsanity.com/paper/2502.20332/) (2025)
- [Emergence in non-neural models: grokking modular arithmetic via average gradient outer product](https://mlsanity.com/paper/2407.20199/) (2024)
- [Minimum Description Length Recurrent Neural Networks](https://mlsanity.com/paper/2111.00600/) (2021)
- [The Complexity Dynamics of Grokking](https://mlsanity.com/paper/2412.09810/) (2024)
- [Lossless data compression by large models](https://mlsanity.com/paper/2407.07723/) (2024)
- [Grokking Beyond the Euclidean Norm of Model Parameters](https://mlsanity.com/paper/2506.05718/) (2025)
- [Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexity](https://mlsanity.com/paper/2310.17247/) (2023)
- [Grokking at the Edge of Linear Separability](https://mlsanity.com/paper/2410.04489/) (2024)
- [Towards a unified and verified understanding of group-operation networks](https://mlsanity.com/paper/2410.07476/) (2024)
- [Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptation](https://mlsanity.com/paper/2506.00288/) (2025)
- [Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory](https://mlsanity.com/paper/2510.12077/) (2025)
- [From An LLM Swarm To A PDDL-Empowered HIVE: Planning Self-Executed Instructions In A Multi-Modal Jungle](https://mlsanity.com/paper/2412.12839/) (2024)

## Built on by (most cited first, 1 of 1)

- [Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics](https://mlsanity.com/paper/2605.20441/) (2026)

## Abstract (arXiv)

Grokking in modular arithmetic has established itself as the quintessential fruit fly experiment, serving as a critical domain for investigating the mechanistic origins of model generalization. Despite its significance, existing research remains narrowly focused on specific local circuits or optimization tuning, largely overlooking the global structural evolution that fundamentally drives this phenomenon. We propose that grokking originates from a spontaneous simplification of internal model structures governed by the principle of parsimony. We integrate causal, spectral, and algorithmic complexity measures alongside Singular Learning Theory to reveal that the transition from memorization to generalization corresponds to the physical collapse of redundant manifolds and deep information compression, offering a novel perspective for understanding the mechanisms of model overfitting and generalization.