Network Automatic Pruning: Start NAP and Take a Nap
Not on the map. Why?
Abstract (arXiv)
Network pruning can significantly reduce the computation and memory footprint of large neural networks. To achieve a good trade-off between model size and performance, popular pruning techniques usually rely on hand-crafted heuristics and require manually setting the compression ratio for each layer. This process is typically time-consuming and requires expert knowledge to achieve good results. In this paper, we propose NAP, a unified and automatic pruning framework for both fine-grained and structured pruning. It can find out unimportant components of a network and automatically decide appropriate compression ratios for different layers, based on a theoretically sound criterion. Towards this goal, NAP uses an efficient approximation of the Hessian for evaluating the importances of components, based on a Kronecker-factored Approximate Curvature method. Despite its simpleness to use, NAP outperforms previous pruning methods by large margins. For fine-grained pruning, NAP can compress AlexNet and VGG16 by 25x, and ResNet-50 by 6.7x without loss in accuracy on ImageNet. For structured pruning (e.g. channel pruning), it can reduce flops of VGG16 by 5.4x and ResNet-50 by 2.3x with only 1% accuracy drop. More importantly, this method is almost free from hyper-parameter tuning and requires no expert knowledge. You can start NAP and then take a nap!
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 17 Jan 2021.
Builds on
30 of 41 references.
References · 30
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
- 2016Pruning Filters for Efficient ConvNetscited by 1,129
- 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
- 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
- 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
- 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
- 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
18 more
- 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
- 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
- 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
- 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
- 2015Tensorizing Neural Networkscited by 246
- 2016Dynamic Network Surgery for Efficient DNNscited by 233
- 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
- 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 212
- 2016Trained Ternary Quantizationcited by 203
- 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
- 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
- 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
- 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
- 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 119
- 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
- 2016Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1cited by 102
- 2017Noisy Natural Gradient as Variational Inferencecited by 94
- 2018AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inferencecited by 77
Built on by
Citing papers per year: 2022: 1.
1 paper.