# Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks arXiv: https://arxiv.org/abs/2206.01278 ยท HTML: https://mlsanity.com/paper/2206.01278/ ## Key facts - Cited by 21 papers in the map (data through 14 Oct 2026). - Builds on 22 of its 33 references that are in the map. - First posted on arXiv 2 Jun 2022. ## Builds on (most cited first, 22 of 22) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018) - [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015) - [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016) - [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018) - [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019) - [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017) - [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018) - [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021) - [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019) - [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018) - [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021) - [The Lottery Ticket Hypothesis for Pre-trained BERT Networks](https://mlsanity.com/paper/2007.12223/) (2020) - [Uniform convergence may be unable to explain generalization in deep learning](https://mlsanity.com/paper/1902.04742/) (2019) - [Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel](https://mlsanity.com/paper/2010.15110/) (2020) - [Deep Learning Through the Lens of Example Difficulty](https://mlsanity.com/paper/2106.09647/) (2021) - [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020) - [When Do Curricula Work?](https://mlsanity.com/paper/2012.03107/) (2020) - [Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP](https://mlsanity.com/paper/1906.02768/) (2019) - [Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets](https://mlsanity.com/paper/1906.06247/) (2019) - [Winning Lottery Tickets in Deep Generative Models](https://mlsanity.com/paper/2010.02350/) (2020) - [On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.01648/) (2021) ## Built on by (most cited first, 21 of 21) - [Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?](https://mlsanity.com/paper/2210.03044/) (2022) - [Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning](https://mlsanity.com/paper/2311.13613/) (2023) - [Can we achieve robustness from data alone?](https://mlsanity.com/paper/2207.11727/) (2022) - [Masks, Signs, And Learning Rate Rewinding](https://mlsanity.com/paper/2402.19262/) (2024) - [Simultaneous linear connectivity of neural networks modulo permutation](https://mlsanity.com/paper/2404.06498/) (2024) - [Speculative Coreset Selection for Task-Specific Fine-tuning](https://mlsanity.com/paper/2410.01296/) (2024) - [Robust Active Distillation](https://mlsanity.com/paper/2210.01213/) (2022) - [HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularization](https://mlsanity.com/paper/2308.07163/) (2023) - [Evolution-aware VAriance (EVA) Coreset Selection for Medical Image Classification](https://mlsanity.com/paper/2406.05677/) (2024) - [On the special role of class-selective neurons in early training](https://mlsanity.com/paper/2305.17409/) (2023) - [When Layers Play the Lottery, all Tickets Win at Initialization](https://mlsanity.com/paper/2301.10835/) (2023) - [Demystifying Data Organization for Enhanced LLM Training](https://mlsanity.com/paper/2605.30334/) (2026) - [Pruning at Initialization -- A Sketching Perspective](https://mlsanity.com/paper/2305.17559/) (2023) - [Linear Mode Connectivity in Sparse Neural Networks](https://mlsanity.com/paper/2310.18769/) (2023) - [Distilled Pruning: Using Synthetic Data to Win the Lottery](https://mlsanity.com/paper/2307.03364/) (2023) - [On How Iterative Magnitude Pruning Discovers Local Receptive Fields in Fully Connected Neural Networks](https://mlsanity.com/paper/2412.06545/) (2024) - [Co(ve)rtex: ML Models as storage channels and their (mis-)applications](https://mlsanity.com/paper/2307.08811/) (2023) - [Insights into the Lottery Ticket Hypothesis and Iterative Magnitude Pruning](https://mlsanity.com/paper/2403.15022/) (2024) - [Chasing Better Deep Image Priors between Over- and Under-parameterization](https://mlsanity.com/paper/2410.24187/) (2024) - [Finding Stable Subnetworks at Initialization with Dataset Distillation](https://mlsanity.com/paper/2503.17905/) (2025) - [Toy Combinatorial Interpretability Models Reveal Lottery Tickets in Early Feature Space](https://mlsanity.com/paper/2605.17704/) (2026) ## Abstract (arXiv) A striking observation about iterative magnitude pruning (IMP; Frankle et al. 2020) is that unicodex2014 after just a few hundred steps of dense training unicodex2014 the method can find a sparse sub-network that can be trained to the same accuracy as the dense network. However, the same does not hold at step 0, i.e. random initialization. In this work, we seek to understand how this early phase of pre-training leads to a good initialization for IMP both through the lens of the data distribution and the loss landscape geometry. Empirically we observe that, holding the number of pre-training iterations constant, training on a small fraction of (randomly chosen) data suffices to obtain an equally good initialization for IMP. We additionally observe that by pre-training only on "easy" training data, we can decrease the number of steps necessary to find a good initialization for IMP compared to training on the full dataset or a randomly chosen subset. Finally, we identify novel properties of the loss landscape of dense networks that are predictive of IMP performance, showing in particular that more examples being linearly mode connected in the dense network correlates well with good initializations for IMP. Combined, these results provide new insight into the role played by the early phase training in IMP.