# NISP: Pruning Networks using Neuron Importance Score Propagation

arXiv: https://arxiv.org/abs/1711.05908 · HTML: https://mlsanity.com/paper/1711.05908/

## Key facts

- Cited by 228 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2017.

## Built on by (most cited first, 30 of 228)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [Learned Token Pruning for Transformers](https://mlsanity.com/paper/2107.00910/) (2021)
- [Group Fisher Pruning for Practical Network Compression](https://mlsanity.com/paper/2108.00708/) (2021)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge](https://mlsanity.com/paper/2110.14032/) (2021)
- [Unified Visual Transformer Compression](https://mlsanity.com/paper/2203.08243/) (2022)
- [CHIP: CHannel Independence-based Pruning for Compact Neural Networks](https://mlsanity.com/paper/2110.13981/) (2021)
- [Convolutional Neural Network Pruning with Structural Redundancy Reduction](https://mlsanity.com/paper/2104.03438/) (2021)
- [Pufferfish: Communication-efficient Models At No Extra Cost](https://mlsanity.com/paper/2103.03936/) (2021)
- [Revisiting Random Channel Pruning for Neural Network Compression](https://mlsanity.com/paper/2205.05676/) (2022)
- [Pruning vs Quantization: Which is Better?](https://mlsanity.com/paper/2307.02973/) (2023)
- [Hessian-Aware Pruning and Optimal Neural Implant](https://mlsanity.com/paper/2101.08940/) (2021)
- [ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity](https://mlsanity.com/paper/2208.02507/) (2022)
- [The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks](https://mlsanity.com/paper/2203.04466/) (2022)
- [Towards Compact CNNs via Collaborative Compression](https://mlsanity.com/paper/2105.11228/) (2021)
- [CHEX: CHannel EXploration for CNN Model Compression](https://mlsanity.com/paper/2203.15794/) (2022)
- [CondenseNet V2: Sparse Feature Reactivation for Deep Networks](https://mlsanity.com/paper/2104.04382/) (2021)
- [Achieving on-Mobile Real-Time Super-Resolution with Neural Architecture and Pruning Search](https://mlsanity.com/paper/2108.08910/) (2021)
- [Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition](https://mlsanity.com/paper/2107.11442/) (2021)
- [Applications and Techniques for Fast Machine Learning in Science](https://mlsanity.com/paper/2110.13041/) (2021)
- [DKM: Differentiable K-Means Clustering Layer for Neural Network Compression](https://mlsanity.com/paper/2108.12659/) (2021)
- [Pruning's Effect on Generalization Through the Lens of Training and Regularization](https://mlsanity.com/paper/2210.13738/) (2022)
- [FairGRAPE: Fairness-aware GRAdient Pruning mEthod for Face Attribute Classification](https://mlsanity.com/paper/2207.10888/) (2022)
- [What Matters In The Structured Pruning of Generative Language Models?](https://mlsanity.com/paper/2302.03773/) (2023)
- [NTK-SAP: Improving neural network pruning by aligning training dynamics](https://mlsanity.com/paper/2304.02840/) (2023)
- [DECORE: Deep Compression with Reinforcement Learning](https://mlsanity.com/paper/2106.06091/) (2021)
- [Exploring Token Pruning in Vision State Space Models](https://mlsanity.com/paper/2409.18962/) (2024)
- [Effective Model Sparsification by Scheduled Grow-and-Prune Methods](https://mlsanity.com/paper/2106.09857/) (2021)
- [An Information-Theoretic Justification for Model Pruning](https://mlsanity.com/paper/2102.08329/) (2021)

## Abstract (arXiv)

To reduce the significant redundancy in deep Convolutional Neural Networks (CNNs), most existing methods prune neurons by only considering statistics of an individual layer or two consecutive layers (e.g., prune one layer to minimize the reconstruction error of the next layer), ignoring the effect of error propagation in deep networks. In contrast, we argue that it is essential to prune neurons in the entire neuron network jointly based on a unified goal: minimizing the reconstruction error of important responses in the "final response layer" (FRL), which is the second-to-last layer before classification, for a pruned network to retrain its predictive power. Specifically, we apply feature ranking techniques to measure the importance of each neuron in the FRL, and formulate network pruning as a binary integer optimization problem and derive a closed-form solution to it for pruning neurons in earlier layers. Based on our theoretical analysis, we propose the Neuron Importance Score Propagation (NISP) algorithm to propagate the importance scores of final responses to every neuron in the network. The CNN is pruned by removing neurons with least importance, and then fine-tuned to retain its predictive power. NISP is evaluated on several datasets with multiple CNN models and demonstrated to achieve significant acceleration and compression with negligible accuracy loss.