mlsanityData to 7 Oct 2026

Rethinking the Value of Network Pruning

arXiv 1810.05270 ↗First posted 11 Oct 2018cs.LG cs.CV stat.MLcited by 595

Not on the map. Why?

Abstract (arXiv)

Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy. In this work, we make several surprising observations which contradict common beliefs. For all state-of-the-art structured pruning algorithms we examined, fine-tuning a pruned model only gives comparable or worse performance than training that model with randomly initialized weights. For pruning algorithms which assume a predefined target network architecture, one can get rid of the full pipeline and directly train the target network from scratch. Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm. Our results suggest the need for more careful baseline evaluations in future research on structured pruning methods. We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin 2019), and find that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization.

Key facts. Cited by 595 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 166; 2022: 115; 2023: 115; 2024: 98; 2025: 68; 2026: 33.

595 papers.

Sort:

By area:

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  3. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  4. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  5. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  6. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  7. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  8. 2025Training Language Models to Reason Efficientlycited by 242
  9. 2021Accelerating Sparse Deep Neural Networkscited by 213
  10. 2023Model Sparsity Can Simplify Machine Unlearningcited by 206
  11. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  12. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
38 more
  1. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  2. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  3. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
  4. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  5. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  6. 2021A Unified Lottery Ticket Hypothesis for Graph Neural Networkscited by 118
  7. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  8. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  9. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  10. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
  11. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  12. 2021Dynamic Slimmable Networkcited by 83
  13. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  14. 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 77
  15. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 77
  16. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  17. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  18. 2021Convolutional Neural Network Pruning with Structural Redundancy Reductioncited by 73
  19. 2022PD-Quant: Post-Training Quantization based on Prediction Difference Metriccited by 73
  20. 2021Global Vision Transformer Pruning with Hessian-Aware Saliencycited by 69
  21. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  22. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 64
  23. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  24. 2021A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustnesscited by 62
  25. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  26. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  27. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  28. 2021Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimizationcited by 57
  29. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  30. 2021Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Networkcited by 55
  31. 2021Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?cited by 52
  32. 2021A Survey on Green Deep Learningcited by 52
  33. 2024ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruningcited by 51
  34. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  35. 2022Revisiting Random Channel Pruning for Neural Network Compressioncited by 50
  36. 2021Learning Language Specific Sub-network for Multilingual Machine Translationcited by 48
  37. 2021Meta-Learning Sparse Implicit Neural Representationscited by 48
  38. 2023EVC: Towards Real-Time Neural Image Compression with Mask Decaycited by 46

50 most cited of 595 All 595 (JSON)