Optimizing Learning Rate Schedules for Iterative Pruning of Deep Neural Networks
Not on the map. Why?
Abstract (arXiv)
The importance of learning rate (LR) schedules on network pruning has been observed in a few recent works. As an example, Frankle and Carbin (2019) highlighted that winning tickets (i.e., accuracy preserving subnetworks) can not be found without applying a LR warmup schedule and Renda, Frankle and Carbin (2020) demonstrated that rewinding the LR to its initial state at the end of each pruning cycle improves performance. In this paper, we go one step further by first providing a theoretical justification for the surprising effect of LR schedules. Next, we propose a LR schedule for network pruning called SILO, which stands for S-shaped Improved Learning rate Optimization. The advantages of SILO over existing state-of-the-art (SOTA) LR schedules are two-fold: (i) SILO has a strong theoretical motivation and dynamically adjusts the LR during pruning to improve generalization. Specifically, SILO increases the LR upper bound (max_lr) in an S-shape. This leads to an improvement of 2% - 4% in extensive experiments with various types of networks (e.g., Vision Transformers, ResNet) on popular datasets such as ImageNet, CIFAR-10/100. (ii) In addition to the strong theoretical motivation, SILO is empirically optimal in the sense of matching an Oracle, which exhaustively searches for the optimal value of max_lr via grid search. We find that SILO is able to precisely adjust the value of max_lr to be within the Oracle optimized interval, resulting in performance competitive with the Oracle with significantly lower complexity.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 55 of its 93 references that are in the map. First posted on arXiv 9 Dec 2022.
Builds on
55 of 93 references.
References · 55
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2016Densely Connected Convolutional Networkscited by 6,691
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
- 2018DARTS: Differentiable Architecture Searchcited by 1,782
38 more
- 2016Pruning Filters for Efficient ConvNetscited by 1,129
- 2016An overview of gradient descent optimization algorithmscited by 1,021
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
- 2015Cyclical Learning Rates for Training Neural Networkscited by 627
- 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
- 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
- 2018Rethinking the Value of Network Pruningcited by 595
- 2020What is the State of Neural Network Pruning?cited by 578
- 2019Importance Estimation for Neural Network Pruningcited by 531
- 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
- 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
- 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
- 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 291
- 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
- 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
- 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
- 2013Predicting Parameters in Deep Learningcited by 223
- 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 220
- 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 206
- 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
- 2021VOLO: Vision Outlooker for Visual Recognitioncited by 164
- 2020Layer-adaptive sparsity for the Magnitude-based Pruningcited by 122
- 2020APQ: Joint Search for Network Architecture, Pruning and Quantization Policycited by 98
- 2018Faster gaze prediction with dense networks and Fisher pruningcited by 92
- 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
- 2019Towards Efficient Model Compression via Learned Global Rankingcited by 75
- 2019Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLPcited by 73
- 2021Convolutional Neural Network Pruning with Structural Redundancy Reductioncited by 73
- 2019Pruning from Scratchcited by 72
- 2020Good Subnetworks Provably Exist: Pruning via Greedy Forward Selectioncited by 70
- 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 67
- 2020Lookahead: A Far-Sighted Alternative of Magnitude-based Pruningcited by 61
- 2019Neural Network Pruning with Residual-Connections and Limited-Datacited by 52
- 2022Revisiting Random Channel Pruning for Neural Network Compressioncited by 50
- 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
- 2022DropNet: Reducing Neural Network Complexity via Iterative Pruningcited by 29
- 2019Sparse Transfer Learning via Winning Lottery Ticketscited by 27
50 most cited of 55 All 55 (JSON)
Built on by
Not cited yet.