# ResKD: Residual-Guided Knowledge Distillation

arXiv: https://arxiv.org/abs/2006.04719 · HTML: https://mlsanity.com/paper/2006.04719/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2020.

## Built on by (most cited first, 25 of 25)

- [Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better](https://mlsanity.com/paper/2108.07969/) (2021)
- [Student Network Learning via Evolutionary Knowledge Distillation](https://mlsanity.com/paper/2103.13811/) (2021)
- [Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again](https://mlsanity.com/paper/2210.04427/) (2022)
- [Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2503.16707/) (2025)
- [Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey](https://mlsanity.com/paper/2405.01725/) (2024)
- [Dual Discriminator Adversarial Distillation for Data-free Model Compression](https://mlsanity.com/paper/2104.05382/) (2021)
- [CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective](https://mlsanity.com/paper/2404.14109/) (2024)
- [Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detection](https://mlsanity.com/paper/2308.14286/) (2023)
- [Enhanced Gradient for Differentiable Architecture Search](https://mlsanity.com/paper/2103.12529/) (2021)
- [Self-Distribution Binary Neural Networks](https://mlsanity.com/paper/2103.02394/) (2021)
- [Deep Class Incremental Learning from Decentralized Data](https://mlsanity.com/paper/2203.05984/) (2022)
- [Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatch](https://mlsanity.com/paper/2405.13078/) (2024)
- [ConaCLIP: Exploring Distillation of Fully-Connected Knowledge Interaction Graph for Lightweight Text-Image Retrieval](https://mlsanity.com/paper/2305.17652/) (2023)
- [Hilbert Distillation for Cross-Dimensionality Networks](https://mlsanity.com/paper/2211.04031/) (2022)
- [Knowledge Distillation from Single to Multi Labels: an Empirical Study](https://mlsanity.com/paper/2303.08360/) (2023)
- [Dual-Forward Path Teacher Knowledge Distillation: Bridging the Capacity Gap Between Teacher and Student](https://mlsanity.com/paper/2506.18244/) (2025)
- [Learning Knowledge Representation with Meta Knowledge Distillation for Single Image Super-Resolution](https://mlsanity.com/paper/2207.08356/) (2022)
- [UHKD: A Unified Framework for Heterogeneous Knowledge Distillation via Frequency-Domain Representations](https://mlsanity.com/paper/2510.24116/) (2025)
- [Reducing Capacity Gap in Knowledge Distillation with Review Mechanism for Crowd Counting](https://mlsanity.com/paper/2206.05475/) (2022)
- [IR2Net: Information Restriction and Information Recovery for Accurate Binary Neural Networks](https://mlsanity.com/paper/2210.02637/) (2022)
- [Federated Semi-Supervised Learning with Annotation Heterogeneity](https://mlsanity.com/paper/2303.02445/) (2023)
- [AgentPose: Progressive Distribution Alignment via Feature Agent for Human Pose Distillation](https://mlsanity.com/paper/2501.08088/) (2025)
- [Expandable Residual Approximation for Knowledge Distillation](https://mlsanity.com/paper/2508.16050/) (2025)
- [Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation](https://mlsanity.com/paper/2609.13199/) (2026)
- [Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinement](https://mlsanity.com/paper/2609.19964/) (2026)

## Abstract (arXiv)

Knowledge distillation, aimed at transferring the knowledge from a heavy teacher network to a lightweight student network, has emerged as a promising technique for compressing neural networks. However, due to the capacity gap between the heavy teacher and the lightweight student, there still exists a significant performance gap between them. In this paper, we see knowledge distillation in a fresh light, using the knowledge gap, or the residual, between a teacher and a student as guidance to train a much more lightweight student, called a res-student. We combine the student and the res-student into a new student, where the res-student rectifies the errors of the former student. Such a residual-guided process can be repeated until the user strikes the balance between accuracy and cost. At inference time, we propose a sample-adaptive strategy to decide which res-students are not necessary for each sample, which can save computational cost. Experimental results show that we achieve competitive performance with 18.04%, 23.14%, 53.59%, and 56.86% of the teachers' computational costs on the CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet datasets. Finally, we do thorough theoretical and empirical analysis for our method.