# Importance Estimation for Neural Network Pruning

arXiv: https://arxiv.org/abs/1906.10771 · HTML: https://mlsanity.com/paper/1906.10771/

## Key facts

- Cited by 541 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2019.

## Built on by (most cited first, 30 of 541)

- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.10512/) (2023)
- [A Survey on Model Compression for Large Language Models](https://mlsanity.com/paper/2308.07633/) (2023)
- [Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment](https://mlsanity.com/paper/2312.12148/) (2023)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Language model compression with weighted low-rank factorization](https://mlsanity.com/paper/2207.00112/) (2022)
- [Structural Pruning for Diffusion Models](https://mlsanity.com/paper/2305.10924/) (2023)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [Chasing Sparsity in Vision Transformers: An End-to-End Exploration](https://mlsanity.com/paper/2106.04533/) (2021)
- [EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention](https://mlsanity.com/paper/2305.07027/) (2023)
- [Structured Pruning Learns Compact and Accurate Models](https://mlsanity.com/paper/2204.00408/) (2022)
- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [Dataset Pruning: Reducing Training Data by Examining Generalization Influence](https://mlsanity.com/paper/2205.09329/) (2022)
- [Group Fisher Pruning for Practical Network Compression](https://mlsanity.com/paper/2108.00708/) (2021)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training](https://mlsanity.com/paper/2102.02887/) (2021)
- [Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentation](https://mlsanity.com/paper/2103.14962/) (2021)
- [LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation](https://mlsanity.com/paper/2306.11222/) (2023)
- [Sparse Training via Boosting Pruning Plasticity with Neuroregeneration](https://mlsanity.com/paper/2106.10404/) (2021)
- [Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods](https://mlsanity.com/paper/2402.02834/) (2024)
- [PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance](https://mlsanity.com/paper/2206.12562/) (2022)
- [CHIP: CHannel Independence-based Pruning for Compact Neural Networks](https://mlsanity.com/paper/2110.13981/) (2021)
- [Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.08566/) (2023)
- [Global Vision Transformer Pruning with Hessian-Aware Saliency](https://mlsanity.com/paper/2110.04869/) (2021)
- [Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifiers](https://mlsanity.com/paper/2301.11578/) (2023)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [Advancing Model Pruning via Bi-level Optimization](https://mlsanity.com/paper/2210.04092/) (2022)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)

## Abstract (arXiv)

Structural pruning of neural network parameters reduces computation, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the first and second-order Taylor expansions to approximate a filter's contribution. Both methods scale consistently across any network layer without requiring per-layer sensitivity analysis and can be applied to any kind of layer, including skip connections. For modern networks trained on ImageNet, we measured experimentally a high (>93%) correlation between the contribution computed by our methods and a reliable estimate of the true importance. Pruning with the proposed methods leads to an improvement over state-of-the-art in terms of accuracy, FLOPs, and parameter reduction. On ResNet-101, we achieve a 40% FLOPS reduction by removing 30% of the parameters, with a loss of 0.02% in the top-1 accuracy on ImageNet. Code is available at https://github.com/NVlabs/Taylor_pruning.