Considering Layerwise Importance in the Lottery Ticket Hypothesis
Not on the map. Why?
Abstract (arXiv)
The Lottery Ticket Hypothesis (LTH) showed that by iteratively training a model, removing connections with the lowest global weight magnitude and rewinding the remaining connections, sparse networks can be extracted. This global comparison removes context information between connections within a layer. Here we study means for recovering some of this layer distributional context and generalise the LTH to consider weight importance values rather than global weight magnitudes. We find that given a repeatable training procedure, applying different importance metrics leads to distinct performant lottery tickets with little overlapping connections. This strongly suggests that lottery tickets are not unique
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 22 Feb 2023.
Builds on
42 of 55 references.
References · 42
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
- 2016Pruning Filters for Efficient ConvNetscited by 1,129
- 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
- 2018Rethinking the Value of Network Pruningcited by 595
- 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
- 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
30 more
- 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
- 2019The State of Sparsity in Deep Neural Networkscited by 471
- 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
- 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
- 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
- 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
- 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
- 2013Predicting Parameters in Deep Learningcited by 223
- 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
- 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 206
- 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
- 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
- 2020Layer-adaptive sparsity for the Magnitude-based Pruningcited by 122
- 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
- 2021A Unified Lottery Ticket Hypothesis for Graph Neural Networkscited by 118
- 2019One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizerscited by 102
- 2020The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Modelscited by 91
- 2020Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficientcited by 81
- 2020Gradient Flow in Sparse Neural Networks and How Lottery Tickets Wincited by 77
- 2020Logarithmic Pruning is All You Needcited by 71
- 2019The Difficulty of Training Sparse Neural Networkscited by 62
- 2021Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Networkcited by 55
- 2019Sparse Transfer Learning via Winning Lottery Ticketscited by 27
- 2020Winning Lottery Tickets in Deep Generative Modelscited by 25
- 2019Evaluating Lottery Tickets Under Distributional Shiftscited by 23
- 2021The Elastic Lottery Ticket Hypothesiscited by 23
- 2021Pruning Randomly Initialized Neural Networks with Iterative Randomizationcited by 20
- 2022A General Framework For Proving The Equivariant Strong Lottery Ticket Hypothesiscited by 19
- 2021Towards Understanding Iterative Magnitude Pruning: Why Lottery Tickets Wincited by 11
- 2020On the Transferability of Winning Tickets in Non-Natural Image Datasetscited by 10
Built on by
Citing papers per year: 2023: 1.
1 paper.