# Few Shot Network Compression via Cross Distillation

arXiv: https://arxiv.org/abs/1911.09450 · HTML: https://mlsanity.com/paper/1911.09450/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2019.

## Built on by (most cited first, 26 of 26)

- [BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction](https://mlsanity.com/paper/2102.05426/) (2021)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2203.09811/) (2022)
- [CrossKD: Cross-Head Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2306.11369/) (2023)
- [Towards Efficient Post-training Quantization of Pre-trained Language Models](https://mlsanity.com/paper/2109.15082/) (2021)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm](https://mlsanity.com/paper/2110.08190/) (2021)
- [Function-Consistent Feature Distillation](https://mlsanity.com/paper/2304.11832/) (2023)
- [A Unified Framework for Soft Threshold Pruning](https://mlsanity.com/paper/2302.13019/) (2023)
- [Practical Network Acceleration with Tiny Sets](https://mlsanity.com/paper/2202.07861/) (2022)
- [Dual Discriminator Adversarial Distillation for Data-free Model Compression](https://mlsanity.com/paper/2104.05382/) (2021)
- [Dense Vision Transformer Compression with Few Samples](https://mlsanity.com/paper/2403.18708/) (2024)
- [Compressing Models with Few Samples: Mimicking then Replacing](https://mlsanity.com/paper/2201.02620/) (2022)
- [Few-Shot Learning of Compact Models via Task-Specific Meta Distillation](https://mlsanity.com/paper/2210.09922/) (2022)
- [COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation](https://mlsanity.com/paper/2507.07580/) (2025)
- [Deeplite Neutrino: An End-to-End Framework for Constrained Deep Learning Model Optimization](https://mlsanity.com/paper/2101.04073/) (2021)
- [Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)](https://mlsanity.com/paper/2110.00684/) (2021)
- [MLink: Linking Black-Box Models from Multiple Domains for Collaborative Inference](https://mlsanity.com/paper/2209.13883/) (2022)
- [Practical Network Acceleration with Tiny Sets: Hypothesis, Theory, and Algorithm](https://mlsanity.com/paper/2303.00972/) (2023)
- [Boomerang Distillation Enables Zero-Shot Model Size Interpolation](https://mlsanity.com/paper/2510.05064/) (2025)
- [Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Experts](https://mlsanity.com/paper/2402.14035/) (2024)
- [Conditional Generative Data-free Knowledge Distillation](https://mlsanity.com/paper/2112.15358/) (2021)
- [Distillation from heterogeneous unlabeled collections](https://mlsanity.com/paper/2201.06507/) (2022)
- [StableKD: Breaking Inter-block Optimization Entanglement for Stable Knowledge Distillation](https://mlsanity.com/paper/2312.13223/) (2023)
- [PRG: Prompt-Based Distillation Without Annotation via Proxy Relational Graph](https://mlsanity.com/paper/2408.12248/) (2024)
- [Compressing Model with Few Class-Imbalance Samples: An Out-of-Distribution Expedition](https://mlsanity.com/paper/2502.05832/) (2025)

## Abstract (arXiv)

Model compression has been widely adopted to obtain light-weighted deep neural networks. Most prevalent methods, however, require fine-tuning with sufficient training data to ensure accuracy, which could be challenged by privacy and security issues. As a compromise between privacy and performance, in this paper we investigate few shot network compression: given few samples per class, how can we effectively compress the network with negligible performance drop? The core challenge of few shot network compression lies in high estimation errors from the original network during inference, since the compressed network can easily over-fits on the few training instances. The estimation errors could propagate and accumulate layer-wisely and finally deteriorate the network output. To address the problem, we propose cross distillation, a novel layer-wise knowledge distillation approach. By interweaving hidden layers of teacher and student network, layer-wisely accumulated estimation errors can be effectively reduced.The proposed method offers a general framework compatible with prevalent network compression techniques such as pruning. Extensive experiments on benchmark datasets demonstrate that cross distillation can significantly improve the student network's accuracy when only a few training instances are available.