Data to 7 Oct 2026

Importance Estimation for Neural Network Pruning

arXiv 1906.10771 ↗First posted 25 Jun 2019cs.LG cs.CV stat.MLcited by 531

Not on the map. Why?

Abstract (arXiv)

Structural pruning of neural network parameters reduces computation, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the first and second-order Taylor expansions to approximate a filter's contribution. Both methods scale consistently across any network layer without requiring per-layer sensitivity analysis and can be applied to any kind of layer, including skip connections. For modern networks trained on ImageNet, we measured experimentally a high (>93%) correlation between the contribution computed by our methods and a reliable estimate of the true importance. Pruning with the proposed methods leads to an improvement over state-of-the-art in terms of accuracy, FLOPs, and parameter reduction. On ResNet-101, we achieve a 40% FLOPS reduction by removing 30% of the parameters, with a loss of 0.02% in the top-1 accuracy on ImageNet. Code is available at https://github.com/NVlabs/Taylor_pruning.

Key facts. Cited by 531 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 76; 2022: 66; 2023: 120; 2024: 99; 2025: 99; 2026: 71.

531 papers.

Sort:

By area:

2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
2023A Survey on Model Compression for Large Language Modelscited by 263
2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
2021Accelerating Sparse Deep Neural Networkscited by 213
2023DepGraph: Towards Any Structural Pruningcited by 202
2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
2023Structural Pruning for Diffusion Modelscited by 193
2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
2022Language model compression with weighted low-rank factorizationcited by 192
2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172

38 more

2022Structured Pruning Learns Compact and Accurate Modelscited by 167
2022A Fast Post-Training Pruning Framework for Transformerscited by 161
2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
2021Group Fisher Pruning for Practical Network Compressioncited by 130
2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
2021Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentationcited by 107
2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
2021CHIP: CHannel Independence-based Pruning for Compact Neural Networkscited by 76
2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
2021Global Vision Transformer Pruning with Hessian-Aware Saliencycited by 69
2023Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifierscited by 66
2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
2022Advancing Model Pruning via Bi-level Optimizationcited by 55
2023LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuningcited by 53
2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
2022Revisiting Random Channel Pruning for Neural Network Compressioncited by 50
2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 49
2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 49
2021FIDNet: LiDAR Point Cloud Semantic Segmentation with Fully Interpolation Decodingcited by 46
2023EVC: Towards Real-Time Neural Image Compression with Mask Decaycited by 46
2022ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsitycited by 45
2023IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuningcited by 45
2021Manifold Regularized Dynamic Network Pruningcited by 43
2024SlimGPT: Layer-wise Structured Pruning for Large Language Modelscited by 43
2021Recent Advances on Neural Network Pruning at Initializationcited by 41
2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
2022Structural Pruning via Latency-Saliency Knapsackcited by 40
2021When to Prune? A Policy towards Early Structural Pruningcited by 38
2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptationcited by 37
2021Towards Compact CNNs via Collaborative Compressioncited by 36
2022CHEX: CHannel EXploration for CNN Model Compressioncited by 36
2021Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decompositioncited by 33

50 most cited of 531 All 531 (JSON)