# Knapsack Pruning with Inner Distillation

arXiv: https://arxiv.org/abs/2002.08258 · HTML: https://mlsanity.com/paper/2002.08258/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2020.

## Built on by (most cited first, 20 of 20)

- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [Structural Pruning via Latency-Saliency Knapsack](https://mlsanity.com/paper/2210.06659/) (2022)
- [SPDY: Accurate Pruning with Speedup Guarantees](https://mlsanity.com/paper/2201.13096/) (2022)
- [What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers](https://mlsanity.com/paper/2302.11874/) (2023)
- [HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search](https://mlsanity.com/paper/2102.11646/) (2021)
- [Coaching a Teachable Student](https://mlsanity.com/paper/2306.10014/) (2023)
- [Hierarchical Selective Classification](https://mlsanity.com/paper/2405.11533/) (2024)
- [LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging](https://mlsanity.com/paper/2406.12837/) (2024)
- [Compact and Optimal Deep Learning with Recurrent Parameter Generators](https://mlsanity.com/paper/2107.07110/) (2021)
- [Pruning Compact ConvNets for Efficient Inference](https://mlsanity.com/paper/2301.04502/) (2023)
- [Enhanced Sparsification via Stimulative Training](https://mlsanity.com/paper/2403.06417/) (2024)
- [Measure Twice, Cut Once: Quantifying Bias and Fairness in Deep Neural Networks](https://mlsanity.com/paper/2110.04397/) (2021)
- [Which models are innately best at uncertainty estimation?](https://mlsanity.com/paper/2206.02152/) (2022)
- [L-GreCo: Layerwise-Adaptive Gradient Compression for Efficient and Accurate Deep Learning](https://mlsanity.com/paper/2210.17357/) (2022)
- [Efficient Latency-Aware CNN Depth Compression via Two-Stage Dynamic Programming](https://mlsanity.com/paper/2301.12187/) (2023)
- [Securing Neural Networks with Knapsack Optimization](https://mlsanity.com/paper/2304.10442/) (2023)
- [Supervised Domain Adaptation for Recognizing Retinal Diseases from Wide-Field Fundus Images](https://mlsanity.com/paper/2305.08078/) (2023)
- [Convolutional Prompting for Broad-Domain Retinal Vessel Segmentation](https://mlsanity.com/paper/2412.18089/) (2024)
- [Exploiting Features with Split-and-Share Module](https://mlsanity.com/paper/2108.04500/) (2021)
- [Learning Sparse Neural Networks with Identity Layers](https://mlsanity.com/paper/2307.07389/) (2023)

## Abstract (arXiv)

Neural network pruning reduces the computational cost of an over-parameterized network to improve its efficiency. Popular methods vary from ℓ1-norm sparsification to Neural Architecture Search (NAS). In this work, we propose a novel pruning method that optimizes the final accuracy of the pruned network and distills knowledge from the over-parameterized parent network's inner layers. To enable this approach, we formulate the network pruning as a Knapsack Problem which optimizes the trade-off between the importance of neurons and their associated computational cost. Then we prune the network channels while maintaining the high-level structure of the network. The pruned network is fine-tuned under the supervision of the parent network using its inner network knowledge, a technique we refer to as the Inner Knowledge Distillation. Our method leads to state-of-the-art pruning results on ImageNet, CIFAR-10 and CIFAR-100 using ResNet backbones. To prune complex network structures such as convolutions with skip-links and depth-wise convolutions, we propose a block grouping approach to cope with these structures. Through this we produce compact architectures with the same FLOPs as EfficientNet-B0 and MobileNetV3 but with higher accuracy, by 1% and 0.3% respectively on ImageNet, and faster runtime on GPU.