mlsanityData to 7 Oct 2026

To prune, or not to prune: exploring the efficacy of pruning for model compression

arXiv 1710.01878 ↗First posted 5 Oct 2017stat.ML cs.LGcited by 596

Not on the map. Why?

Abstract (arXiv)

Model pruning seeks to induce sparsity in a deep neural network's various connection matrices, thereby reducing the number of nonzero-valued parameters in the model. Recent reports (Han et al., 2015; Narang et al., 2017) prune deep networks at the cost of only a marginal loss in accuracy and achieve a sizable reduction in model size. This hints at the possibility that the baseline models in these experiments are perhaps severely over-parameterized at the outset and a viable alternative for model compression might be to simply reduce the number of hidden units while maintaining the model's dense connection structure, exposing a similar trade-off in model size and accuracy. We investigate these two distinct paths for model compression within the context of energy-efficient inference in resource-constrained environments and propose a new gradual pruning technique that is simple and straightforward to apply across a variety of models/datasets with minimal tuning and can be seamlessly incorporated within the training process. We compare the accuracy of large, but pruned models (large-sparse) and their smaller, but dense (small-dense) counterparts with identical memory footprint. Across a broad range of neural network architectures (deep CNNs, stacked LSTM, and seq2seq LSTM models), we find large-sparse models to consistently outperform small-dense models and achieve up to 10x reduction in number of non-zero parameters with minimal loss in accuracy.

Key facts. Cited by 596 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 131; 2022: 120; 2023: 116; 2024: 110; 2025: 83; 2026: 36.

596 papers.

Sort:

By area:

  1. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  3. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  4. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  5. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  6. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  7. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  8. 2021Accelerating Sparse Deep Neural Networkscited by 213
  9. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  10. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  11. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  12. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
38 more
  1. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  2. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 116
  3. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  4. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  5. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  6. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  7. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 101
  8. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
  9. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  10. 2021QuantumNAS: Noise-Adaptive Search for Robust Quantum Circuitscited by 97
  11. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  12. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  13. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  14. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  15. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 77
  16. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  17. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 75
  18. 2022PD-Quant: Post-Training Quantization based on Prediction Difference Metriccited by 73
  19. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  20. 2021SoK: How Robust is Image Classification Deep Neural Network Watermarking? (Extended Version)cited by 67
  21. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  22. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 64
  23. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  24. 2021A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustnesscited by 62
  25. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  26. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  27. 2021A Comprehensive Survey on Hardware-Aware Neural Architecture Searchcited by 60
  28. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  29. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  30. 2021TT-Rec: Tensor Train Compression for Deep Learning Recommendation Modelscited by 58
  31. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 58
  32. 2022Fingerprinting Deep Neural Networks Globally via Universal Adversarial Perturbationscited by 58
  33. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  34. 2021CDFI: Compression-Driven Network Design for Frame Interpolationcited by 55
  35. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  36. 2022Advancing Model Pruning via Bi-level Optimizationcited by 55
  37. 2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
  38. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 54

50 most cited of 596 All 596 (JSON)