# Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?

arXiv: https://arxiv.org/abs/2210.03044 · HTML: https://mlsanity.com/paper/2210.03044/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 6 Oct 2022.

## Builds on (most cited first, 27 of 27)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [Comparing Rewinding and Fine-tuning in Neural Network Pruning](https://mlsanity.com/paper/2003.02389/) (2020)
- [What's Hidden in a Randomly Weighted Neural Network?](https://mlsanity.com/paper/1911.13299/) (2019)
- [Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask](https://mlsanity.com/paper/1905.01067/) (2019)
- [Pruning Neural Networks at Initialization: Why are We Missing the Mark?](https://mlsanity.com/paper/2009.08576/) (2020)
- [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020)
- [Winning the Lottery with Continuous Sparsification](https://mlsanity.com/paper/1912.04427/) (2019)
- [Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win](https://mlsanity.com/paper/2010.03533/) (2020)
- [Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot](https://mlsanity.com/paper/2009.11094/) (2020)
- [Rare Gems: Finding Lottery Tickets at Initialization](https://mlsanity.com/paper/2202.12002/) (2022)
- [How many degrees of freedom do we need to train deep networks: a loss landscape perspective](https://mlsanity.com/paper/2107.05802/) (2021)
- [Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks](https://mlsanity.com/paper/2206.01278/) (2022)
- [Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models](https://mlsanity.com/paper/2205.12694/) (2022)

## Built on by (most cited first, 30 of 38)

- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression](https://mlsanity.com/paper/2306.15063/) (2023)
- [Break It Down: Evidence for Structural Compositionality in Neural Networks](https://mlsanity.com/paper/2301.10884/) (2023)
- [Approaching Deep Learning through the Spectral Dynamics of Weights](https://mlsanity.com/paper/2408.11804/) (2024)
- [A Survey of Lottery Ticket Hypothesis](https://mlsanity.com/paper/2403.04861/) (2024)
- [CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Models](https://mlsanity.com/paper/2210.09223/) (2022)
- [Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging](https://mlsanity.com/paper/2306.16788/) (2023)
- [Pruning Large Language Models with Semi-Structural Adaptive Sparse Training](https://mlsanity.com/paper/2407.20584/) (2024)
- [Masks, Signs, And Learning Rate Rewinding](https://mlsanity.com/paper/2402.19262/) (2024)
- [Mask in the Mirror: Implicit Sparsification](https://mlsanity.com/paper/2408.09966/) (2024)
- [Simultaneous linear connectivity of neural networks modulo permutation](https://mlsanity.com/paper/2404.06498/) (2024)
- [How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective](https://mlsanity.com/paper/2306.05857/) (2023)
- [Maestro: Uncovering Low-Rank Structures via Trainable Decomposition](https://mlsanity.com/paper/2308.14929/) (2023)
- [No Free Prune: Information-Theoretic Barriers to Pruning at Initialization](https://mlsanity.com/paper/2402.01089/) (2024)
- [Sign-In to the Lottery: Reparameterizing Sparse Training From Scratch](https://mlsanity.com/paper/2504.12801/) (2025)
- [Sparse Weight Averaging with Multiple Particles for Iterative Magnitude Pruning](https://mlsanity.com/paper/2305.14852/) (2023)
- [Quantifying lottery tickets under label noise: accuracy, calibration, and complexity](https://mlsanity.com/paper/2306.12190/) (2023)
- [Optimal Eye Surgeon: Finding Image Priors through Sparse Generators at Initialization](https://mlsanity.com/paper/2406.05288/) (2024)
- [The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws](https://mlsanity.com/paper/2501.12486/) (2025)
- [Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry](https://mlsanity.com/paper/2505.05143/) (2025)
- [Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks](https://mlsanity.com/paper/2607.11875/) (2026)
- [Distilled Pruning: Using Synthetic Data to Win the Lottery](https://mlsanity.com/paper/2307.03364/) (2023)
- [Iterative Magnitude Pruning as a Renormalisation Group: A Study in The Context of The Lottery Ticket Hypothesis](https://mlsanity.com/paper/2308.03128/) (2023)
- [Pruning neural network models for gene regulatory dynamics using data and domain knowledge](https://mlsanity.com/paper/2403.04805/) (2024)
- [Cyclic Sparse Training: Is it Enough?](https://mlsanity.com/paper/2406.02773/) (2024)
- [Find A Winning Sign: Sign Is All We Need to Win the Lottery](https://mlsanity.com/paper/2504.05357/) (2025)
- [CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Models](https://mlsanity.com/paper/2509.25996/) (2025)
- [Routing the Lottery: Adaptive Subnetworks for Heterogeneous Data](https://mlsanity.com/paper/2601.22141/) (2026)
- [It's Not a Lottery, It's a Race: Understanding How Gradient Descent Adapts the Network's Capacity to the Task](https://mlsanity.com/paper/2602.04832/) (2026)

## Abstract (arXiv)

Modern deep learning involves training costly, highly overparameterized networks, thus motivating the search for sparser networks that can still be trained to the same accuracy as the full network (i.e. matching). Iterative magnitude pruning (IMP) is a state of the art algorithm that can find such highly sparse matching subnetworks, known as winning tickets. IMP operates by iterative cycles of training, masking smallest magnitude weights, rewinding back to an early training point, and repeating. Despite its simplicity, the underlying principles for when and how IMP finds winning tickets remain elusive. In particular, what useful information does an IMP mask found at the end of training convey to a rewound network near the beginning of training? How does SGD allow the network to extract this information? And why is iterative pruning needed? We develop answers in terms of the geometry of the error landscape. First, we find thatunicodex2014at higher sparsitiesunicodex2014pairs of pruned networks at successive pruning iterations are connected by a linear path with zero error barrier if and only if they are matching. This indicates that masks found at the end of training convey the identity of an axial subspace that intersects a desired linearly connected mode of a matching sublevel set. Second, we show SGD can exploit this information due to a strong form of robustness: it can return to this mode despite strong perturbations early in training. Third, we show how the flatness of the error landscape at the end of training determines a limit on the fraction of weights that can be pruned at each iteration of IMP. Finally, we show that the role of retraining in IMP is to find a network with new small weights to prune. Overall, these results make progress toward demystifying the existence of winning tickets by revealing the fundamental role of error landscape geometry.