mlsanityData to 14 Oct 2026

One Shot vs. Iterative: Rethinking Pruning Strategies for Model Compression

arXiv 2508.13836 ↗First posted 19 Aug 2025cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Pruning is a core technique for compressing neural networks to improve computational efficiency. This process is typically approached in two ways: one-shot pruning, which involves a single pass of training and pruning, and iterative pruning, where pruning is performed over multiple cycles for potentially finer network refinement. Although iterative pruning has historically seen broader adoption, this preference is often assumed rather than rigorously tested. Our study presents one of the first systematic and comprehensive comparisons of these methods, providing rigorous definitions, benchmarking both across structured and unstructured settings, and applying different pruning criteria and modalities. We find that each method has specific advantages: one-shot pruning proves more effective at lower pruning ratios, while iterative pruning performs better at higher ratios. Building on these findings, we advocate for patience-based pruning and introduce a hybrid approach that can outperform traditional methods in certain scenarios, providing valuable insights for practitioners selecting a pruning strategy tailored to their goals and constraints. Source code is available at https://github.com/janumiko/pruning-benchmark.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 44 of its 54 references that are in the map. First posted on arXiv 19 Aug 2025.

Builds on

44 of 54 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2016Densely Connected Convolutional Networkscited by 6,720
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  10. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  11. 2018Rethinking the Value of Network Pruningcited by 599
  12. 2019Importance Estimation for Neural Network Pruningcited by 541
32 more
  1. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  2. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  3. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  4. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  5. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 378
  6. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 367
  7. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 295
  8. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 292
  9. 2020MCUNet: Tiny Deep Learning on IoT Devicescited by 273
  10. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 240
  11. 2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 227
  12. 2023DepGraph: Towards Any Structural Pruningcited by 203
  13. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  14. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  15. 2021Group Fisher Pruning for Practical Network Compressioncited by 131
  16. 2018Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layerscited by 112
  17. 2019Provable Filter Pruning for Efficient Neural Networkscited by 98
  18. 2018Faster gaze prediction with dense networks and Fisher pruningcited by 94
  19. 2021CHIP: CHannel Independence-based Pruning for Compact Neural Networkscited by 78
  20. 2019Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structurecited by 76
  21. 2021Convolutional Neural Network Pruning with Structural Redundancy Reductioncited by 73
  22. 2020Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compressioncited by 72
  23. 2020DHP: Differentiable Meta Pruning via HyperNetworkscited by 57
  24. 2019Neural Network Pruning with Residual-Connections and Limited-Datacited by 52
  25. 2022Revisiting Random Channel Pruning for Neural Network Compressioncited by 50
  26. 2016Pruning Convolutional Neural Networks for Resource Efficient Inferencecited by 48
  27. 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
  28. 2019OICSR: Out-In-Channel Sparsity Regularization for Compact Deep Neural Networkscited by 18
  29. 2019SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networkscited by 17
  30. 2020The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architecturescited by 13
  31. 2020Dirichlet Pruning for Neural Network Compressioncited by 3
  32. 2019Neuron ranking -- an informed way to condense convolutional neural networks architecturecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025PosterCopilot: Toward Layout Reasoning and Controllable Editing for Professional Graphic Designcited by 6
  2. 2025Hardware-Aware YOLO Compression for Low-Power Edge AI on STM32U5 for Weeds Detection in Digital Agriculturecited by 1
  3. 2026Artificial Structure Function Search: Preserving Artificial Functional Connectivity for Structured Pruningcited by 0