mlsanityData to 7 Oct 2026

Practical Network Acceleration with Tiny Sets

arXiv 2202.07861 ↗First posted 16 Feb 2022cs.CV cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Due to data privacy issues, accelerating networks with tiny training sets has become a critical need in practice. Previous methods mainly adopt filter-level pruning to accelerate networks with scarce training samples. In this paper, we reveal that dropping blocks is a fundamentally superior approach in this scenario. It enjoys a higher acceleration ratio and results in a better latency-accuracy performance under the few-shot setting. To choose which blocks to drop, we propose a new concept namely recoverability to measure the difficulty of recovering the compressed network. Our recoverability is efficient and effective for choosing which blocks to drop. Finally, we propose an algorithm named PRACTISE to accelerate networks using only tiny sets of training images. PRACTISE outperforms previous methods by a significant margin. For 22% latency reduction, PRACTISE surpasses previous methods by on average 7% on ImageNet-1k. It also enjoys high generalization ability, working well under data-free or out-of-domain data settings, too. Our code is at https://github.com/DoctorKey/Practise.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 35 of its 43 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

35 of 43 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  10. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  11. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  12. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
23 more
  1. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  2. 2018Rethinking the Value of Network Pruningcited by 595
  3. 2019Importance Estimation for Neural Network Pruningcited by 531
  4. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  5. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
  6. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
  7. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
  8. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
  9. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
  10. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
  11. 2020ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgettingcited by 85
  12. 2016Places: An Image Database for Deep Scene Understandingcited by 76
  13. 2021Diversifying Sample Generation for Accurate Data-Free Quantizationcited by 63
  14. 2019Neural Network Pruning with Residual-Connections and Limited-Datacited by 52
  15. 2020Discrimination-aware Network Pruning for Deep Model Compressioncited by 50
  16. 2021Zero-shot Adversarial Quantizationcited by 44
  17. 2020Distilling Knowledge by Mimicking Featurescited by 34
  18. 2020MixMix: All You Need for Data-Free Compression Are Feature and Data Mixingcited by 29
  19. 2019Few Shot Network Compression via Cross Distillationcited by 26
  20. 2019DBP: Discrimination Based Block-Level Pruning for Deep Model Accelerationcited by 24
  21. 2018Network Decoupling: From Regular to Depthwise Separable Convolutionscited by 17
  22. 2022Compressing Models with Few Samples: Mimicking then Replacingcited by 8
  23. 2020Joint Multi-Dimension Pruning via Numerical Gradient Updatecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 3; 2026: 1.

9 papers.

  1. 2023EVC: Towards Real-Time Neural Image Compression with Mask Decaycited by 46
  2. 2024Dense Vision Transformer Compression with Few Samplescited by 8
  3. 2025Stratified Knowledge-Density Super-Network for Scalable Vision Transformerscited by 3
  4. 2023Practical Network Acceleration with Tiny Sets: Hypothesis, Theory, and Algorithmcited by 2
  5. 2025BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compressioncited by 2
  6. 2026Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compressioncited by 2
  7. 2024TOAST: Transformer Optimization using Adaptive and Simple Transformationscited by 1
  8. 2024TT-MPD: Test Time Model Pruning and Distillationcited by 0
  9. 2025Compressing Model with Few Class-Imbalance Samples: An Out-of-Distribution Expeditioncited by 0