Adaptive Activation-based Structured Pruning
Not on the map. Why?
Abstract (arXiv)
Pruning is a promising approach to compress complex deep learning models in order to deploy them on resource-constrained edge devices. However, many existing pruning solutions are based on unstructured pruning, which yields models that cannot efficiently run on commodity hardware and require users to manually explore and tune the pruning process, which is time-consuming and often leads to sub-optimal results. To address these limitations, this paper presents an adaptive, activation-based, structured pruning approach to automatically and efficiently generate small, accurate, and hardware-efficient models that meet user requirements. First, it proposes iterative structured pruning using activation-based attention feature maps to effectively identify and prune unimportant filters. Then, it proposes adaptive pruning policies for automatically meeting the pruning objectives of accuracy-critical, memory-constrained, and latency-sensitive tasks. A comprehensive evaluation shows that the proposed method can substantially outperform the state-of-the-art structured pruning works on CIFAR-10 and ImageNet datasets. For example, on ResNet-56 with CIFAR-10, without any accuracy drop, our method achieves the largest parameter reduction (79.11%), outperforming the related works by 22.81% to 66.07%, and the largest FLOPs reduction (70.13%), outperforming the related works by 14.13% to 26.53%.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 29 of its 37 references that are in the map. First posted on arXiv 21 Jan 2022.
Builds on
29 of 37 references.
References · 29
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2016Pruning Filters for Efficient ConvNetscited by 1,129
- 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
- 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
- 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
- 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
- 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
- 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
- 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 313
- 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 291
17 more
- 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
- 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
- 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
- 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
- 2019Towards Optimal Structured CNN Pruning via Generative Adversarial Learningcited by 145
- 2019Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networkscited by 118
- 2019EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasiscited by 98
- 2019Provable Filter Pruning for Efficient Neural Networkscited by 98
- 2018Faster gaze prediction with dense networks and Fisher pruningcited by 92
- 2020SCOP: Scientific Control for Reliable Neural Network Pruningcited by 92
- 2020Operation-Aware Soft Channel Pruning using Differentiable Maskscited by 74
- 2019Pruning from Scratchcited by 72
- 2020Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compressioncited by 72
- 2019DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measurescited by 56
- 2017N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learningcited by 33
- 2021GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarizationcited by 29
- 2017Efficient Architecture Search by Network Transformationcited by 9
Built on by
Citing papers per year: 2023: 1; 2024: 0; 2025: 2.
3 papers.