# Sparse Transfer Learning via Winning Lottery Tickets

arXiv: https://arxiv.org/abs/1905.07785 · HTML: https://mlsanity.com/paper/1905.07785/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2019.

## Built on by (most cited first, 27 of 27)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?](https://mlsanity.com/paper/2107.00166/) (2021)
- [How Well Do Sparse Imagenet Models Transfer?](https://mlsanity.com/paper/2111.13445/) (2021)
- [Efficient Lottery Ticket Finding: Less Data is More](https://mlsanity.com/paper/2106.03225/) (2021)
- [GANs Can Play Lottery Tickets Too](https://mlsanity.com/paper/2106.00134/) (2021)
- [Exploring Lottery Ticket Hypothesis in Spiking Neural Networks](https://mlsanity.com/paper/2207.01382/) (2022)
- [On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.01648/) (2021)
- [A Survey of Lottery Ticket Hypothesis](https://mlsanity.com/paper/2403.04861/) (2024)
- [Rethinking Graph Lottery Tickets: Graph Sparsity Matters](https://mlsanity.com/paper/2305.02190/) (2023)
- [Towards Understanding Iterative Magnitude Pruning: Why Lottery Tickets Win](https://mlsanity.com/paper/2106.06955/) (2021)
- [Differentially Private Video Activity Recognition](https://mlsanity.com/paper/2306.15742/) (2023)
- [Universality of Winning Tickets: A Renormalization Group Perspective](https://mlsanity.com/paper/2110.03210/) (2021)
- [On the Opportunities of Green Computing: A Survey](https://mlsanity.com/paper/2311.00447/) (2023)
- [Data-Efficient Double-Win Lottery Tickets from Robust Pre-training](https://mlsanity.com/paper/2206.04762/) (2022)
- [Uncovering the Hidden Cost of Model Compression](https://mlsanity.com/paper/2308.14969/) (2023)
- [S-Cyc: A Learning Rate Schedule for Iterative Pruning of ReLU-based Networks](https://mlsanity.com/paper/2110.08764/) (2021)
- [Towards Compute-Optimal Transfer Learning](https://mlsanity.com/paper/2304.13164/) (2023)
- [Optimal Eye Surgeon: Finding Image Priors through Sparse Generators at Initialization](https://mlsanity.com/paper/2406.05288/) (2024)
- [Considering Layerwise Importance in the Lottery Ticket Hypothesis](https://mlsanity.com/paper/2302.11244/) (2023)
- [On How Iterative Magnitude Pruning Discovers Local Receptive Fields in Fully Connected Neural Networks](https://mlsanity.com/paper/2412.06545/) (2024)
- [Optimizing Learning Rate Schedules for Iterative Pruning of Deep Neural Networks](https://mlsanity.com/paper/2212.06144/) (2022)
- [AP: Selective Activation for De-sparsifying Pruned Neural Networks](https://mlsanity.com/paper/2212.06145/) (2022)
- [Transferability of Winning Lottery Tickets in Neural Network Differential Equation Solvers](https://mlsanity.com/paper/2306.09863/) (2023)
- [What to Do When Your Discrete Optimization Is the Size of a Neural Network?](https://mlsanity.com/paper/2402.10339/) (2024)
- [Beyond Transfer Accuracy: Mechanism-Guided Controlled Adaptation for Low-Resource Languages](https://mlsanity.com/paper/2601.08146/) (2026)
- [Man, Machine, and Mathematics](https://mlsanity.com/paper/2604.27052/) (2026)

## Abstract (arXiv)

The recently proposed Lottery Ticket Hypothesis of Frankle and Carbin (2019) suggests that the performance of over-parameterized deep networks is due to the random initialization seeding the network with a small fraction of favorable weights. These weights retain their dominant status throughout training -- in a very real sense, this sub-network "won the lottery" during initialization. The authors find sub-networks via unstructured magnitude pruning with 85-95% of parameters removed that train to the same accuracy as the original network at a similar speed, which they call winning tickets. In this paper, we extend the Lottery Ticket Hypothesis to a variety of transfer learning tasks. We show that sparse sub-networks with approximately 90-95% of weights removed achieve (and often exceed) the accuracy of the original dense network in several realistic settings. We experimentally validate this by transferring the sparse representation found via pruning on CIFAR-10 to SmallNORB and FashionMNIST for object recognition tasks.