mlsanityData to 7 Oct 2026

Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask

arXiv 1905.01067 ↗First posted 3 May 2019cs.LG cs.CV stat.MLcited by 206

Not on the map. Why?

Abstract (arXiv)

The recent "Lottery Ticket Hypothesis" paper by Frankle & Carbin showed that a simple approach to creating sparse networks (keeping the large weights) results in models that are trainable from scratch, but only when starting from the same initial weights. The performance of these networks often exceeds the performance of the non-sparse base model, but for reasons that were not well understood. In this paper we study the three critical components of the Lottery Ticket (LT) algorithm, showing that each may be varied significantly without impacting the overall results. Ablating these factors leads to new insights for why LT networks perform as well as they do. We show why setting weights to zero is important, how signs are all you need to make the reinitialized network train, and why masking behaves like training. Finally, we discover the existence of Supermasks, masks that can be applied to an untrained, randomly initialized network to produce a model with performance far better than chance (86% on MNIST, 41% on CIFAR-10).

Key facts. Cited by 206 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 52; 2022: 57; 2023: 41; 2024: 29; 2025: 18; 2026: 9.

206 papers.

Sort:

By area:

  1. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  2. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  3. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  4. 2023Modular Deep Learningcited by 82
  5. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  6. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  7. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  8. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  9. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  10. 2021Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Networkcited by 55
  11. 2021Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?cited by 52
  12. 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
38 more
  1. 2021Efficient Lottery Ticket Finding: Less Data is Morecited by 44
  2. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  3. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  4. 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
  5. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  6. 2022Fortuitous Forgetting in Connectionist Networkscited by 39
  7. 2022Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?cited by 38
  8. 2022On the Soft-Subnetwork for Few-shot Class Incremental Learningcited by 37
  9. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  10. 2021Ps and Qs: Quantization-aware pruning for efficient low latency neural network inferencecited by 31
  11. 2021Selfish Sparse RNN Trainingcited by 26
  12. 2022Why Random Pruning Is All We Need to Start Sparsecited by 26
  13. 2022Exploring Lottery Ticket Hypothesis in Spiking Neural Networkscited by 25
  14. 2021The Elastic Lottery Ticket Hypothesiscited by 23
  15. 2023Rethinking Graph Lottery Tickets: Graph Sparsity Matterscited by 23
  16. 2024A Survey of Lottery Ticket Hypothesiscited by 23
  17. 2022Most Activation Functions Can Win the Lottery Without Excessive Depthcited by 21
  18. 2022Winning the Lottery Ahead of Time: Efficient Early Network Pruningcited by 21
  19. 2021Pruning Randomly Initialized Neural Networks with Iterative Randomizationcited by 20
  20. 2022Lifelong Reinforcement Learning with Modulating Maskscited by 20
  21. 2021Charged particle tracking via edge-classifying interaction networkscited by 19
  22. 2021Knowledge Evolution in Neural Networkscited by 18
  23. 2022Modular and On-demand Bias Mitigation with Attribute-Removal Subnetworkscited by 18
  24. 2022Comprehensive Graph Gradual Pruning for Sparse Training in Graph Neural Networkscited by 18
  25. 2023One Network, Many Masks: Towards More Parameter-Efficient Transfer Learningcited by 18
  26. 2024Algorithmic Capabilities of Random Transformerscited by 17
  27. 2022Convolutional and Residual Networks Provably Contain Lottery Ticketscited by 16
  28. 2023Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Trainingcited by 16
  29. 2021Truly Sparse Neural Networks at Scalecited by 15
  30. 2021A Probabilistic Approach to Neural Network Pruningcited by 15
  31. 2024Masks, Signs, And Learning Rate Rewindingcited by 15
  32. 2021Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networkscited by 14
  33. 2022Provable and Efficient Continual Representation Learningcited by 14
  34. 2022DiSparse: Disentangled Sparsification for Multitask Model Compressioncited by 14
  35. 2022You Can Have Better Graph Neural Networks by Not Training Weights at All: Finding Untrained GNNs Ticketscited by 14
  36. 2021Plant 'n' Seek: Can You Find the Winning Ticket?cited by 13
  37. 2022HARFE: Hard-Ridge Random Feature Expansioncited by 12
  38. 2022Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Surveycited by 12

50 most cited of 206 All 206 (JSON)