# Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networks arXiv: https://arxiv.org/abs/2310.19470 ยท HTML: https://mlsanity.com/paper/2310.19470/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Builds on 29 of its 43 references that are in the map. - First posted on arXiv 30 Oct 2023. ## Builds on (most cited first, 29 of 29) - [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018) - [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023) - [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018) - [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018) - [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019) - [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022) - [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020) - [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020) - [What's Hidden in a Randomly Weighted Neural Network?](https://mlsanity.com/paper/1911.13299/) (2019) - [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022) - [The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks](https://mlsanity.com/paper/2306.17844/) (2023) - [Omnigrok: Grokking Beyond Algorithmic Data](https://mlsanity.com/paper/2210.01117/) (2022) - [Explaining grokking through circuit efficiency](https://mlsanity.com/paper/2309.02390/) (2023) - [Grokking as the Transition from Lazy to Rich Training Dynamics](https://mlsanity.com/paper/2310.06110/) (2023) - [The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon](https://mlsanity.com/paper/2206.04817/) (2022) - [Grokking modular arithmetic](https://mlsanity.com/paper/2301.02679/) (2023) - [A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworks](https://mlsanity.com/paper/2303.11873/) (2023) - [Pruning from Scratch](https://mlsanity.com/paper/1909.12579/) (2019) - [Graph Structure of Neural Networks](https://mlsanity.com/paper/2007.06559/) (2020) - [Grokking of Hierarchical Structure in Vanilla Transformers](https://mlsanity.com/paper/2305.18741/) (2023) - [Unifying Grokking and Double Descent](https://mlsanity.com/paper/2303.06173/) (2023) - [Grokking as a First Order Phase Transition in Two Layer Networks](https://mlsanity.com/paper/2310.03789/) (2023) - [Towards Learning Convolutions from Scratch](https://mlsanity.com/paper/2007.13657/) (2020) - [Efficient Lottery Ticket Finding: Less Data is More](https://mlsanity.com/paper/2106.03225/) (2021) - [Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn features](https://mlsanity.com/paper/2212.13881/) (2022) - [Grokking as Compression: A Nonlinear Complexity Perspective](https://mlsanity.com/paper/2310.05918/) (2023) - [Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexity](https://mlsanity.com/paper/2310.17247/) (2023) - [Analyzing Lottery Ticket Hypothesis from PAC-Bayesian Theory Perspective](https://mlsanity.com/paper/2205.07320/) (2022) - [Bespoke vs. Pret-`a-Porter Lottery Tickets: Exploiting Mask Similarity for Trainable Sub-Network Finding](https://mlsanity.com/paper/2007.04091/) (2020) ## Built on by (most cited first, 6 of 6) - [Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence](https://mlsanity.com/paper/2505.16694/) (2025) - [A Spin Glass Characterization of Neural Networks](https://mlsanity.com/paper/2508.07397/) (2025) - [Grokking as Structural Inference: Transformers Need Bayesian Lottery Tickets](https://mlsanity.com/paper/2605.15787/) (2026) - [How to Tame Grokking: Representation Geometry as a Control Signal](https://mlsanity.com/paper/2607.11666/) (2026) - [Slower Generalization, Faster Memorization: A Sweet Spot in Algorithmic Learning](https://mlsanity.com/paper/2605.14659/) (2026) - [Matching Object or Relation? Tracing Abstract Reasoning Inside VLMs](https://mlsanity.com/paper/2610.07646/) (2026) ## Abstract (arXiv) Grokking is an intriguing phenomenon of delayed generalization, where neural networks initially memorize training data with perfect accuracy but exhibit poor generalization, subsequently transitioning to a generalizing solution with continued training. While factors such as weight norms and sparsity have been proposed to explain this delayed generalization, the influence of network structure remains underexplored. In this work, we link the grokking phenomenon to the lottery ticket hypothesis to investigate the impact of internal network structures. We demonstrate that utilizing lottery tickets obtained during the generalizing phase (termed grokked tickets) significantly reduces delayed generalization across various tasks, including multiple modular arithmetic operations, polynomial regression, sparse parity, and MNIST classification. Through controlled experiments, we show that the mitigation of delayed generalization is not due solely to reduced weight norms or increased sparsity, but rather to the discovery of good subnetworks. Furthermore, we find that grokked tickets exhibit periodic weight patterns, beneficial graph properties such as increased average path lengths and reduced clustering coefficients, and undergo rapid structural changes that coincide with improvements in generalization. Additionally, pruning techniques like the edge-popup algorithm can identify these effective structures without modifying the weights, thereby transforming memorizing networks into generalizing ones. These results underscore the novel insight that structural exploration plays a pivotal role in understanding grokking. The implementation code can be accessed via this link: https://github.com/gouki510/Grokking-Tickets.