# Towards Understanding Iterative Magnitude Pruning: Why Lottery Tickets Win

arXiv: https://arxiv.org/abs/2106.06955 · HTML: https://mlsanity.com/paper/2106.06955/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 29 references that are in the map.
- First posted on arXiv 13 Jun 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [An Empirical Model of Large-Batch Training](https://mlsanity.com/paper/1812.06162/) (2018)
- [The Lottery Ticket Hypothesis for Pre-trained BERT Networks](https://mlsanity.com/paper/2007.12223/) (2020)
- [Comparing Rewinding and Fine-tuning in Neural Network Pruning](https://mlsanity.com/paper/2003.02389/) (2020)
- [Sparse Networks from Scratch: Faster Training without Losing Performance](https://mlsanity.com/paper/1907.04840/) (2019)
- [Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask](https://mlsanity.com/paper/1905.01067/) (2019)
- [Pruning Neural Networks at Initialization: Why are We Missing the Mark?](https://mlsanity.com/paper/2009.08576/) (2020)
- [What Do Compressed Deep Neural Networks Forget?](https://mlsanity.com/paper/1911.05248/) (2019)
- [Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks](https://mlsanity.com/paper/1909.11957/) (2019)
- [Soft Threshold Weight Reparameterization for Learnable Sparsity](https://mlsanity.com/paper/2002.03231/) (2020)
- [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020)
- [Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs](https://mlsanity.com/paper/2003.00152/) (2020)
- [One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers](https://mlsanity.com/paper/1906.02773/) (2019)
- [Winning the Lottery with Continuous Sparsification](https://mlsanity.com/paper/1912.04427/) (2019)
- [Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win](https://mlsanity.com/paper/2010.03533/) (2020)
- [Sparse Transfer Learning via Winning Lottery Tickets](https://mlsanity.com/paper/1905.07785/) (2019)

## Built on by (most cited first, 11 of 11)

- [Mask in the Mirror: Implicit Sparsification](https://mlsanity.com/paper/2408.09966/) (2024)
- [Universality of Winning Tickets: A Renormalization Group Perspective](https://mlsanity.com/paper/2110.03210/) (2021)
- [On the Opportunities of Green Computing: A Survey](https://mlsanity.com/paper/2311.00447/) (2023)
- [S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-training](https://mlsanity.com/paper/2409.09099/) (2024)
- [Exploiting Sparsity in Pruned Neural Networks to Optimize Large Model Training](https://mlsanity.com/paper/2302.05045/) (2023)
- [Position: Cracking the Code of Cascading Disparity Towards Marginalized Communities](https://mlsanity.com/paper/2406.01757/) (2024)
- [Lottery Tickets in Evolutionary Optimization: On Sparse Backpropagation-Free Trainability](https://mlsanity.com/paper/2306.00045/) (2023)
- [Considering Layerwise Importance in the Lottery Ticket Hypothesis](https://mlsanity.com/paper/2302.11244/) (2023)
- [Iterative Magnitude Pruning as a Renormalisation Group: A Study in The Context of The Lottery Ticket Hypothesis](https://mlsanity.com/paper/2308.03128/) (2023)
- [Transferability of Winning Lottery Tickets in Neural Network Differential Equation Solvers](https://mlsanity.com/paper/2306.09863/) (2023)
- [Man, Machine, and Mathematics](https://mlsanity.com/paper/2604.27052/) (2026)

## Abstract (arXiv)

The lottery ticket hypothesis states that sparse subnetworks exist in randomly initialized dense networks that can be trained to the same accuracy as the dense network they reside in. However, the subsequent work has failed to replicate this on large-scale models and required rewinding to an early stable state instead of initialization. We show that by using a training method that is stable with respect to linear mode connectivity, large networks can also be entirely rewound to initialization. Our subsequent experiments on common vision tasks give strong credence to the hypothesis in Evci et al. (2020b) that lottery tickets simply retrain to the same regions (although not necessarily to the same basin). These results imply that existing lottery tickets could not have been found without the preceding dense training by iterative magnitude pruning, raising doubts about the use of the lottery ticket hypothesis.