mlsanityData to 14 Oct 2026

PDP: Parameter-free Differentiable Pruning is All You Need

arXiv 2305.11203 ↗First posted 18 May 2023cs.LG cs.AI cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

DNN pruning is a popular way to reduce the size of a model, improve the inference latency, and minimize the power consumption on DNN accelerators. However, existing approaches might be too complex, expensive or ineffective to apply to a variety of vision/language tasks, DNN architectures and to honor structured pruning constraints. In this paper, we propose an efficient yet effective train-time pruning scheme, Parameter-free Differentiable Pruning (PDP), which offers state-of-the-art qualities in model size, accuracy, and training cost. PDP uses a dynamic function of weights during training to generate soft pruning masks for the weights in a parameter-free manner for a given pruning target. While differentiable, the simplicity and efficiency of PDP make it universal enough to deliver state-of-the-art random/structured/channel pruning results on various vision and natural language tasks. For example, for MobileNet-v1, PDP can achieve 68.2% top-1 ImageNet1k accuracy at 86.6% sparsity, which is 1.7% higher accuracy than those from the state-of-the-art algorithms. Also, PDP yields over 83.1% accuracy on Multi-Genre Natural Language Inference with 90% sparsity for BERT, while the next best from the existing techniques shows 81.5% accuracy. In addition, PDP can be applied to structured pruning, such as N:M pruning and channel pruning. For 1:4 structured pruning of ResNet18, PDP improved the top-1 ImageNet1k accuracy by over 3.6% over the state-of-the-art. For channel pruning of ResNet50, PDP reduced the top-1 ImageNet1k accuracy by 0.6% from the state-of-the-art.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 51 of its 62 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

51 of 62 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  9. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  10. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  11. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  12. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 642
38 more
  1. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  2. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  3. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  4. 2019The State of Sparsity in Deep Neural Networkscited by 476
  5. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  6. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  7. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  8. 2018Model compression via distillation and quantizationcited by 289
  9. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  10. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  11. 2021Accelerating Sparse Deep Neural Networkscited by 223
  12. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 207
  13. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
  14. 2021Block Pruning For Faster Transformerscited by 177
  15. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  16. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 151
  17. 2015Structured Pruning of Deep Convolutional Neural Networkscited by 149
  18. 2019Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networkscited by 148
  19. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  20. 2019Winning the Lottery with Continuous Sparsificationcited by 100
  21. 2021Network Quantization with Element-wise Gradient Scalingcited by 87
  22. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 84
  23. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 76
  24. 2020Operation-Aware Soft Channel Pruning using Differentiable Maskscited by 74
  25. 2020DMCP: Differentiable Markov Channel Pruning for Neural Networkscited by 69
  26. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
  27. 2021FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizescited by 66
  28. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  29. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 59
  30. 2019Discovering Neural Wiringscited by 56
  31. 2020PROFIT: A Novel Training Method for sub-4-bit MobileNet Modelscited by 50
  32. 2020DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocationcited by 47
  33. 2020Differentiable Joint Pruning and Quantization for Hardware Efficiencycited by 46
  34. 2022SPDY: Accurate Pruning with Speedup Guaranteescited by 36
  35. 2018Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutionscited by 33
  36. 2021DKM: Differentiable K-Means Clustering Layer for Neural Network Compressioncited by 31
  37. 2021Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Predictioncited by 24
  38. 2021S2TA: Exploiting Structured Sparsity for Energy-Efficient Mobile CNN Accelerationcited by 19

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 4; 2026: 3.

12 papers.

  1. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  2. 2024Separate, Dynamic and Differentiable (SMART) Pruner for Block/Output Channel Pruning on Computer Vision Taskscited by 1
  3. 2026SimCert: Probabilistic Certification for Behavioral Similarity in Deep Neural Network Compressioncited by 1
  4. 2024SequentialAttention++ for Block Sparsification: Differentiable Pruning Meets Combinatorial Optimizationcited by 0
  5. 2024S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruningcited by 0
  6. 2024A Data-Free Analytical Quantization Scheme for Deep Learning Modelscited by 0
  7. 2025The Effects of Grouped Structural Global Pruning of Vision Transformers on Domain Generalisationcited by 0
  8. 2025Hyperflux: Pruning Reveals Importancecited by 0
  9. 2025MUC-G4: Minimal Unsat Core-Guided Incremental Verification for Deep Neural Network Compressioncited by 0
  10. 2025Unified Start, Personalized End: Progressive Pruning for Efficient 3D Medical Image Segmentationcited by 0
  11. 2026PQuantML: A Tool for End-to-End Hardware-aware Model Compressioncited by 0
  12. 2026HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Modelscited by 0