# Practical Network Acceleration with Tiny Sets

arXiv: https://arxiv.org/abs/2202.07861 · HTML: https://mlsanity.com/paper/2202.07861/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 43 references that are in the map.
- First posted on arXiv 16 Feb 2022.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [ZeroQ: A Novel Zero Shot Quantization Framework](https://mlsanity.com/paper/2001.00281/) (2020)
- [Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon](https://mlsanity.com/paper/1705.07565/) (2017)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [Scalable Methods for 8-bit Training of Neural Networks](https://mlsanity.com/paper/1805.11046/) (2018)
- [ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting](https://mlsanity.com/paper/2007.03260/) (2020)
- [Places: An Image Database for Deep Scene Understanding](https://mlsanity.com/paper/1610.02055/) (2016)
- [Diversifying Sample Generation for Accurate Data-Free Quantization](https://mlsanity.com/paper/2103.01049/) (2021)
- [Neural Network Pruning with Residual-Connections and Limited-Data](https://mlsanity.com/paper/1911.08114/) (2019)
- [Discrimination-aware Network Pruning for Deep Model Compression](https://mlsanity.com/paper/2001.01050/) (2020)
- [Zero-shot Adversarial Quantization](https://mlsanity.com/paper/2103.15263/) (2021)
- [Distilling Knowledge by Mimicking Features](https://mlsanity.com/paper/2011.01424/) (2020)
- [MixMix: All You Need for Data-Free Compression Are Feature and Data Mixing](https://mlsanity.com/paper/2011.09899/) (2020)

## Built on by (most cited first, 10 of 10)

- [EVC: Towards Real-Time Neural Image Compression with Mask Decay](https://mlsanity.com/paper/2302.05071/) (2023)
- [Dense Vision Transformer Compression with Few Samples](https://mlsanity.com/paper/2403.18708/) (2024)
- [TOAST: Transformer Optimization using Adaptive and Simple Transformations](https://mlsanity.com/paper/2410.04941/) (2024)
- [Stratified Knowledge-Density Super-Network for Scalable Vision Transformers](https://mlsanity.com/paper/2511.11683/) (2025)
- [Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compression](https://mlsanity.com/paper/2603.18426/) (2026)
- [Practical Network Acceleration with Tiny Sets: Hypothesis, Theory, and Algorithm](https://mlsanity.com/paper/2303.00972/) (2023)
- [BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression](https://mlsanity.com/paper/2509.25136/) (2025)
- [TT-MPD: Test Time Model Pruning and Distillation](https://mlsanity.com/paper/2412.07114/) (2024)
- [Compressing Model with Few Class-Imbalance Samples: An Out-of-Distribution Expedition](https://mlsanity.com/paper/2502.05832/) (2025)
- [The Functional Structure of Post-Compression Recovery in Low-Rank LLMs](https://mlsanity.com/paper/2610.05504/) (2026)

## Abstract (arXiv)

Due to data privacy issues, accelerating networks with tiny training sets has become a critical need in practice. Previous methods mainly adopt filter-level pruning to accelerate networks with scarce training samples. In this paper, we reveal that dropping blocks is a fundamentally superior approach in this scenario. It enjoys a higher acceleration ratio and results in a better latency-accuracy performance under the few-shot setting. To choose which blocks to drop, we propose a new concept namely recoverability to measure the difficulty of recovering the compressed network. Our recoverability is efficient and effective for choosing which blocks to drop. Finally, we propose an algorithm named PRACTISE to accelerate networks using only tiny sets of training images. PRACTISE outperforms previous methods by a significant margin. For 22% latency reduction, PRACTISE surpasses previous methods by on average 7% on ImageNet-1k. It also enjoys high generalization ability, working well under data-free or out-of-domain data settings, too. Our code is at https://github.com/DoctorKey/Practise.