# Respecting Transfer Gap in Knowledge Distillation arXiv: https://arxiv.org/abs/2210.12787 ยท HTML: https://mlsanity.com/paper/2210.12787/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. - Builds on 61 of its 87 references that are in the map. - First posted on arXiv 23 Oct 2022. ## Placed in - Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/) ## Builds on (most cited first, 30 of 61) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016) - [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018) - [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016) - [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016) - [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016) - [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016) - [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019) - [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014) - [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020) - [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017) - [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018) - [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016) - [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019) - [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019) - [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019) - [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019) - [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020) - [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019) - [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018) - [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019) - [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019) - [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019) - [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019) - [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019) - [Distilling Knowledge via Knowledge Review](https://mlsanity.com/paper/2104.09044/) (2021) - [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019) - [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019) ## Built on by (most cited first, 18 of 18) - [VkD: Improving Knowledge Distillation using Orthogonal Projections](https://mlsanity.com/paper/2403.06213/) (2024) - [DiGeo: Discriminative Geometry-Aware Learning for Generalized Few-Shot Object Detection](https://mlsanity.com/paper/2303.09674/) (2023) - [NormKD: Normalized Logits for Knowledge Distillation](https://mlsanity.com/paper/2308.00520/) (2023) - [CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective](https://mlsanity.com/paper/2404.14109/) (2024) - [AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation](https://mlsanity.com/paper/2403.07030/) (2024) - [Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning](https://mlsanity.com/paper/2507.10348/) (2025) - [DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer](https://mlsanity.com/paper/2505.15133/) (2025) - [Local Dense Logit Relations for Enhanced Knowledge Distillation](https://mlsanity.com/paper/2507.15911/) (2025) - [The Role of Teacher Calibration in Knowledge Distillation](https://mlsanity.com/paper/2508.20224/) (2025) - [Towards Effective Collaborative Learning in Long-Tailed Recognition](https://mlsanity.com/paper/2305.03378/) (2023) - [A Stepwise Distillation Learning Strategy for Non-differentiable Visual Programming Frameworks on Visual Reasoning Tasks](https://mlsanity.com/paper/2309.09809/) (2023) - [Understanding the Effects of Projectors in Knowledge Distillation](https://mlsanity.com/paper/2310.17183/) (2023) - [Asymmetric Decision-Making in Online Knowledge Distillation:Unifying Consensus and Divergence](https://mlsanity.com/paper/2503.06685/) (2025) - [Domain Invariant Learning for Gaussian Processes and Bayesian Exploration](https://mlsanity.com/paper/2312.11318/) (2023) - [Cross-Domain Knowledge Distillation for Low-Resolution Human Pose Estimation](https://mlsanity.com/paper/2405.11448/) (2024) - [Discriminative and Consistent Representation Distillation](https://mlsanity.com/paper/2407.11802/) (2024) - [Revisiting the Relationship between Adversarial and Clean Training: Why Clean Training Can Make Adversarial Training Better](https://mlsanity.com/paper/2504.00038/) (2025) - [Heterogeneous Complementary Distillation](https://mlsanity.com/paper/2511.10942/) (2025) ## Abstract (arXiv) Knowledge distillation (KD) is essentially a process of transferring a teacher model's behavior, e.g., network response, to a student model. The network response serves as additional supervision to formulate the machine domain, which uses the data collected from the human domain as a transfer set. Traditional KD methods hold an underlying assumption that the data collected in both human domain and machine domain are both independent and identically distributed (IID). We point out that this naive assumption is unrealistic and there is indeed a transfer gap between the two domains. Although the gap offers the student model external knowledge from the machine domain, the imbalanced teacher knowledge would make us incorrectly estimate how much to transfer from teacher to student per sample on the non-IID transfer set. To tackle this challenge, we propose Inverse Probability Weighting Distillation (IPWD) that estimates the propensity score of a training sample belonging to the machine domain, and assigns its inverse amount to compensate for under-represented samples. Experiments on CIFAR-100 and ImageNet demonstrate the effectiveness of IPWD for both two-stage distillation and one-stage self-distillation.