# Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup

arXiv: https://arxiv.org/abs/2012.09413 · HTML: https://mlsanity.com/paper/2012.09413/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2020.

## Built on by (most cited first, 22 of 22)

- [Adaptive wavelet distillation from neural networks through interpretations](https://mlsanity.com/paper/2107.09145/) (2021)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [Knowledge Condensation Distillation](https://mlsanity.com/paper/2207.05409/) (2022)
- [Active Data Curation Effectively Distills Large-Scale Multimodal Models](https://mlsanity.com/paper/2411.18674/) (2024)
- [Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Survey](https://mlsanity.com/paper/2208.03197/) (2022)
- [Robust Active Distillation](https://mlsanity.com/paper/2210.01213/) (2022)
- [Data-Free Quantization via Mixed-Precision Compensation without Fine-Tuning](https://mlsanity.com/paper/2307.00498/) (2023)
- [It's All in the Head: Representation Knowledge Distillation through Classifier Sharing](https://mlsanity.com/paper/2201.06945/) (2022)
- [Learning with Less: Knowledge Distillation from Large Language Models via Unlabeled Data](https://mlsanity.com/paper/2411.08028/) (2024)
- [Can Students Beyond The Teacher? Distilling Knowledge from Teacher's Bias](https://mlsanity.com/paper/2412.09874/) (2024)
- [Cache & Distil: Optimising API Calls to Large Language Models](https://mlsanity.com/paper/2310.13561/) (2023)
- [Sentinel-Guided Zero-Shot Learning: A Collaborative Paradigm without Real Data Exposure](https://mlsanity.com/paper/2403.09363/) (2024)
- [Instance Temperature Knowledge Distillation](https://mlsanity.com/paper/2407.00115/) (2024)
- [Absolute Zero-Shot Learning](https://mlsanity.com/paper/2202.11319/) (2022)
- [ACQ: Improving Generative Data-free Quantization Via Attention Correction](https://mlsanity.com/paper/2301.07266/) (2023)
- [Distilling Calibrated Student from an Uncalibrated Teacher](https://mlsanity.com/paper/2302.11472/) (2023)
- [Adaptive Self-Distillation for Minimizing Client Drift in Heterogeneous Federated Learning](https://mlsanity.com/paper/2305.19600/) (2023)
- [Teaching with Uncertainty: Unleashing the Potential of Knowledge Distillation in Object Detection](https://mlsanity.com/paper/2406.06999/) (2024)
- [Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions](https://mlsanity.com/paper/2606.12171/) (2026)
- [Batch Selection and Communication for Active Learning with Edge Labeling](https://mlsanity.com/paper/2311.08053/) (2023)
- [TT-MPD: Test Time Model Pruning and Distillation](https://mlsanity.com/paper/2412.07114/) (2024)
- [Compressing Quaternion Convolutional Neural Networks for Audio Classification](https://mlsanity.com/paper/2510.21388/) (2025)

## Abstract (arXiv)

Knowledge distillation, which involves extracting the "dark knowledge" from a teacher network to guide the learning of a student network, has emerged as an essential technique for model compression and transfer learning. Unlike previous works that focus on the accuracy of student network, here we study a little-explored but important question, i.e., knowledge distillation efficiency. Our goal is to achieve a performance comparable to conventional knowledge distillation with a lower computation cost during training. We show that the UNcertainty-aware mIXup (UNIX) can serve as a clean yet effective solution. The uncertainty sampling strategy is used to evaluate the informativeness of each training sample. Adaptive mixup is applied to uncertain samples to compact knowledge. We further show that the redundancy of conventional knowledge distillation lies in the excessive learning of easy samples. By combining uncertainty and mixup, our approach reduces the redundancy and makes better use of each query to the teacher network. We validate our approach on CIFAR100 and ImageNet. Notably, with only 79% computation cost, we outperform conventional knowledge distillation on CIFAR100 and achieve a comparable result on ImageNet.