Learning Pruned Structure and Weights Simultaneously from Scratch: an Attention based Approach
Not on the map. Why?
Abstract (arXiv)
As a deep learning model typically contains millions of trainable weights, there has been a growing demand for a more efficient network structure with reduced storage space and improved run-time efficiency. Pruning is one of the most popular network compression techniques. In this paper, we propose a novel unstructured pruning pipeline, Attention-based Simultaneous sparse structure and Weight Learning (ASWL). Unlike traditional channel-wise or weight-wise attention mechanism, ASWL proposed an efficient algorithm to calculate the pruning ratio through layer-wise attention for each layer, and both weights for the dense network and the sparse network are tracked so that the pruned structure is simultaneously learned from randomly initialized weights. Our experiments on MNIST, Cifar10, and ImageNet show that ASWL achieves superior pruning results in terms of accuracy, pruning ratio and operating efficiency when compared with state-of-the-art network pruning methods.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 1 Nov 2021.
Builds on
42 of 56 references.
References · 42
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2017Mask R-CNNcited by 7,259
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
- 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
- 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
30 more
- 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
- 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
- 2018Rethinking the Value of Network Pruningcited by 595
- 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 550
- 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
- 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
- 2015Learning Deconvolution Network for Semantic Segmentationcited by 365
- 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
- 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
- 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
- 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
- 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
- 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
- 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
- 2019Towards Optimal Structured CNN Pruning via Generative Adversarial Learningcited by 145
- 2016Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruningcited by 133
- 2020Dynamic Model Pruning with Feedbackcited by 125
- 2017Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutionscited by 93
- 2018AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inferencecited by 77
- 2019Towards Efficient Model Compression via Learned Global Rankingcited by 75
- 2021Convolutional Neural Network Pruning with Structural Redundancy Reductioncited by 73
- 2019Pruning from Scratchcited by 72
- 2020Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpotcited by 66
- 2019AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Ratescited by 49
- 2020Pruning Filter in Filtercited by 46
- 2020EDropout: Energy-Based Dropout and Pruning of Deep Neural Networkscited by 20
- 2020Learned Threshold Pruningcited by 16
- 2017Fine-Pruning: Joint Fine-Tuning and Compression of a Convolutional Network with Bayesian Optimizationcited by 8
- 2018PCAS: Pruning Channels with Attention Statistics for Deep Network Compressioncited by 7
- 2019Attention Based Pruning for Shift Networkscited by 7
Built on by
Citing papers per year: 2022: 1.
1 paper.