mlsanityData to 7 Oct 2026

What is the State of Neural Network Pruning?

arXiv 2003.03033 ↗First posted 6 Mar 2020cs.LG stat.MLcited by 578

Not on the map. Why?

Abstract (arXiv)

Neural network pruning---the task of reducing the size of a network by removing parameters---has been the subject of a great deal of work in recent years. We provide a meta-analysis of the literature, including an overview of approaches to pruning and consistent findings in the literature. After aggregating results across 81 papers and pruning hundreds of models in controlled conditions, our clearest finding is that the community suffers from a lack of standardized benchmarks and metrics. This deficiency is substantial enough that it is hard to compare pruning techniques to one another or determine how much progress the field has made over the past three decades. To address this situation, we identify issues with current practices, suggest concrete remedies, and introduce ShrinkBench, an open-source framework to facilitate standardized evaluations of pruning methods. We use ShrinkBench to compare various pruning techniques and show that its comprehensive evaluation can prevent common pitfalls when comparing pruning methods.

Key facts. Cited by 578 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 101; 2022: 105; 2023: 126; 2024: 113; 2025: 87; 2026: 46.

578 papers.

Sort:

By area:

  1. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  3. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  4. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  5. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  6. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  7. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  8. 2023Model Sparsity Can Simplify Machine Unlearningcited by 206
  9. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  10. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
  11. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  12. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
38 more
  1. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  2. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  3. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  4. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  5. 2021Leveraging Sparse Linear Layers for Debuggable Deep Networkscited by 82
  6. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  7. 2021CHIP: CHannel Independence-based Pruning for Compact Neural Networkscited by 76
  8. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  9. 2022Deep Differentiable Logic Gate Networkscited by 67
  10. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  11. 2023Single-Model and Any-Modality for Video Object Trackingcited by 64
  12. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  13. 2021hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devicescited by 61
  14. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  15. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  16. 2022Advancing Model Pruning via Bi-level Optimizationcited by 55
  17. 2023LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuningcited by 53
  18. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 53
  19. 2024ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruningcited by 51
  20. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  21. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  22. 2022Revisiting Random Channel Pruning for Neural Network Compressioncited by 50
  23. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 50
  24. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 46
  25. 2023Pruning vs Quantization: Which is Better?cited by 45
  26. 2024Trimming the Fat: Efficient Compression of 3D Gaussian Splats through Pruningcited by 45
  27. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  28. 2022Pruning has a disparate impact on model accuracycited by 43
  29. 2024Incremental Residual Concept Bottleneck Modelscited by 43
  30. 2022Membership Inference Attacks and Defenses in Neural Network Pruningcited by 42
  31. 2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 42
  32. 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
  33. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  34. 2023Fast as CHITA: Neural Network Pruning with Combinatorial Optimizationcited by 40
  35. 2025Weight-sparse transformers have interpretable circuitscited by 40
  36. 2022Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?cited by 38
  37. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  38. 2023Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretabilitycited by 38

50 most cited of 578 All 578 (JSON)