# Residual Knowledge Distillation

arXiv: https://arxiv.org/abs/2002.09168 · HTML: https://mlsanity.com/paper/2002.09168/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2020.

## Built on by (most cited first, 11 of 11)

- [Efficient Knowledge Distillation from Model Checkpoints](https://mlsanity.com/paper/2210.06458/) (2022)
- [Distilling Efficient Vision Transformers from CNNs for Semantic Segmentation](https://mlsanity.com/paper/2310.07265/) (2023)
- [Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation](https://mlsanity.com/paper/2102.13088/) (2021)
- [CoCoFL: Communication- and Computation-Aware Federated Learning via Partial NN Freezing and Quantization](https://mlsanity.com/paper/2203.05468/) (2022)
- [Compacting Deep Neural Networks for Internet of Things: Methods and Applications](https://mlsanity.com/paper/2103.11083/) (2021)
- [Learn From the Past: Experience Ensemble Knowledge Distillation](https://mlsanity.com/paper/2202.12488/) (2022)
- [Hilbert Distillation for Cross-Dimensionality Networks](https://mlsanity.com/paper/2211.04031/) (2022)
- [Controlling the Quality of Distillation in Response-Based Network Compression](https://mlsanity.com/paper/2112.10047/) (2021)
- [Distilling Knowledge via Intermediate Classifiers](https://mlsanity.com/paper/2103.00497/) (2021)
- [Reducing Capacity Gap in Knowledge Distillation with Review Mechanism for Crowd Counting](https://mlsanity.com/paper/2206.05475/) (2022)
- [XPERT: Expert Knowledge Transfer for Effective Training of Language Models](https://mlsanity.com/paper/2605.08842/) (2026)

## Abstract (arXiv)

Knowledge distillation (KD) is one of the most potent ways for model compression. The key idea is to transfer the knowledge from a deep teacher model (T) to a shallower student (S). However, existing methods suffer from performance degradation due to the substantial gap between the learning capacities of S and T. To remedy this problem, this work proposes Residual Knowledge Distillation (RKD), which further distills the knowledge by introducing an assistant (A). Specifically, S is trained to mimic the feature maps of T, and A aids this process by learning the residual error between them. In this way, S and A complement with each other to get better knowledge from T. Furthermore, we devise an effective method to derive S and A from a given model without increasing the total computational cost. Extensive experiments show that our approach achieves appealing results on popular classification datasets, CIFAR-100 and ImageNet, surpassing state-of-the-art methods.