mlsanityData to 7 Oct 2026

When Layers Play the Lottery, all Tickets Win at Initialization

arXiv 2301.10835 ↗First posted 25 Jan 2023cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Pruning is a standard technique for reducing the computational cost of deep networks. Many advances in pruning leverage concepts from the Lottery Ticket Hypothesis (LTH). LTH reveals that inside a trained dense network exists sparse subnetworks (tickets) able to achieve similar accuracy (i.e., win the lottery - winning tickets). Pruning at initialization focuses on finding winning tickets without training a dense network. Studies on these concepts share the trend that subnetworks come from weight or filter pruning. In this work, we investigate LTH and pruning at initialization from the lens of layer pruning. First, we confirm the existence of winning tickets when the pruning process removes layers. Leveraged by this observation, we propose to discover these winning tickets at initialization, eliminating the requirement of heavy computational resources for training the initial (over-parameterized) dense network. Extensive experiments show that our winning tickets notably speed up the training phase and reduce up to 51% of carbon emission, an important step towards democratization and green Artificial Intelligence. Beyond computational benefits, our winning tickets exhibit robustness against adversarial and out-of-distribution examples. Finally, we show that our subnetworks easily win the lottery at initialization while tickets from filter removal (the standard structured LTH) hardly become winning tickets.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 36 of its 45 references that are in the map. First posted on arXiv 25 Jan 2023.

Builds on

36 of 45 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  6. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  7. 2016Deep Networks with Stochastic Depthcited by 973
  8. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  9. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  10. 2018Rethinking the Value of Network Pruningcited by 595
  11. 2020What is the State of Neural Network Pruning?cited by 578
  12. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
24 more
  1. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  2. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  3. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  4. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  5. 2021Dynamic Neural Networks: A Surveycited by 405
  6. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 291
  7. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 291
  8. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  9. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  10. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  11. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  12. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  13. 2017Convolutional Networks with Adaptive Inference Graphscited by 147
  14. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  15. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  16. 2020PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruningcited by 75
  17. 2020Progressive Skeletonization: Trimming more fat from a network at initializationcited by 64
  18. 2021A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustnesscited by 62
  19. 2019Neural Network Pruning with Residual-Connections and Limited-Datacited by 52
  20. 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
  21. 2022DropNet: Reducing Neural Network Complexity via Iterative Pruningcited by 29
  22. 2021Lottery Ticket Preserves Weight Correlation: Is It Desirable or Not?cited by 22
  23. 2022Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networkscited by 21
  24. 2020NPAS: A Compiler-aware Framework of Unified Network Pruning and Architecture Search for Beyond Real-Time Mobile Accelerationcited by 16

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2024Effective Layer Pruning Through Similarity Metric Perspectivecited by 6
  2. 2024Layer Pruning with Consensus: A Triple-Win Solutioncited by 0
  3. 2025Pruning Everything, Everywhere, All at Oncecited by 0