# The rise of the lottery heroes: why zero-shot pruning is hard

arXiv: https://arxiv.org/abs/2202.12400 · HTML: https://mlsanity.com/paper/2202.12400/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 24 Feb 2022.

## Builds on (most cited first, 17 of 17)

- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation](https://mlsanity.com/paper/1404.0736/) (2014)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [Proving the Lottery Ticket Hypothesis: Pruning is All You Need](https://mlsanity.com/paper/2002.00585/) (2020)
- [Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks](https://mlsanity.com/paper/1909.11957/) (2019)
- [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020)
- [One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers](https://mlsanity.com/paper/1906.02773/) (2019)
- [Efficient Lottery Ticket Finding: Less Data is More](https://mlsanity.com/paper/2106.03225/) (2021)
- [The Lottery Ticket Hypothesis for Object Recognition](https://mlsanity.com/paper/2012.04643/) (2020)
- [SeReNe: Sensitivity based Regularization of Neurons for Structured Sparsity in Neural Networks](https://mlsanity.com/paper/2102.03773/) (2021)
- [Pruning artificial neural networks: a way to find well-generalizing, high-entropy sharp minima](https://mlsanity.com/paper/2004.14765/) (2020)

## Built on by (most cited first, 7 of 7)

- [Can Unstructured Pruning Reduce the Depth in Deep Neural Networks?](https://mlsanity.com/paper/2308.06619/) (2023)
- [To update or not to update? Neurons at equilibrium in deep models](https://mlsanity.com/paper/2207.09455/) (2022)
- [Towards On-device Learning on the Edge: Ways to Select Neurons to Update under a Budget Constraint](https://mlsanity.com/paper/2312.05282/) (2023)
- [SCoTTi: Save Computation at Training Time with an adaptive framework](https://mlsanity.com/paper/2312.12483/) (2023)
- [Can we avoid Double Descent in Deep Neural Networks?](https://mlsanity.com/paper/2302.13259/) (2023)
- [Debiasing surgeon: fantastic weights and how to find them](https://mlsanity.com/paper/2403.14200/) (2024)
- [Routing the Lottery: Adaptive Subnetworks for Heterogeneous Data](https://mlsanity.com/paper/2601.22141/) (2026)

## Abstract (arXiv)

Recent advances in deep learning optimization showed that just a subset of parameters are really necessary to successfully train a model. Potentially, such a discovery has broad impact from the theory to application; however, it is known that finding these trainable sub-network is a typically costly process. This inhibits practical applications: can the learned sub-graph structures in deep learning models be found at training time? In this work we explore such a possibility, observing and motivating why common approaches typically fail in the extreme scenarios of interest, and proposing an approach which potentially enables training with reduced computational effort. The experiments on either challenging architectures and datasets suggest the algorithmic accessibility over such a computational gain, and in particular a trade-off between accuracy achieved and training complexity deployed emerges.