# What Knowledge Gets Distilled in Knowledge Distillation?

arXiv: https://arxiv.org/abs/2205.16004 · HTML: https://mlsanity.com/paper/2205.16004/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 31 May 2022.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 32)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021)
- [Partial success in closing the gap between human and machine vision](https://mlsanity.com/paper/2106.07411/) (2021)
- [Towards Understanding Knowledge Distillation](https://mlsanity.com/paper/2105.13093/) (2021)
- [Adversarially Robust Distillation](https://mlsanity.com/paper/1905.09747/) (2019)
- [Self-Distillation as Instance-Specific Label Smoothing](https://mlsanity.com/paper/2006.05065/) (2020)

## Built on by (most cited first, 27 of 27)

- [Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation](https://mlsanity.com/paper/2412.08139/) (2024)
- [On student-teacher deviations in distillation: does it pay to disobey?](https://mlsanity.com/paper/2301.12923/) (2023)
- [Provable Weak-to-Strong Generalization via Benign Overfitting](https://mlsanity.com/paper/2410.04638/) (2024)
- [Good Teachers Explain: Explanation-Enhanced Knowledge Distillation](https://mlsanity.com/paper/2402.03119/) (2024)
- [Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimension](https://mlsanity.com/paper/2502.05075/) (2025)
- [CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective](https://mlsanity.com/paper/2404.14109/) (2024)
- [Random Conditioning with Distillation for Data-Efficient Diffusion Model Compression](https://mlsanity.com/paper/2504.02011/) (2025)
- [Zero-Shot Distillation for Image Encoders: How to Make Effective Use of Synthetic Data](https://mlsanity.com/paper/2404.16637/) (2024)
- [Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applications](https://mlsanity.com/paper/2510.27186/) (2025)
- [Group Distributionally Robust Knowledge Distillation](https://mlsanity.com/paper/2311.00476/) (2023)
- [Efficient Verified Machine Unlearning For Distillation](https://mlsanity.com/paper/2503.22539/) (2025)
- [Does Weak-to-strong Generalization Happen under Spurious Correlations?](https://mlsanity.com/paper/2509.24005/) (2025)
- [Distilling Lightweight Domain Experts from Large ML Models by Identifying Relevant Subspaces](https://mlsanity.com/paper/2601.05913/) (2026)
- [Knowledge Distillation Must Account for What It Loses](https://mlsanity.com/paper/2604.25110/) (2026)
- [Weight Averaging Improves Knowledge Distillation under Domain Shift](https://mlsanity.com/paper/2309.11446/) (2023)
- [Stereo-Knowledge Distillation from dpMV to Dual Pixels for Light Field Video Reconstruction](https://mlsanity.com/paper/2405.11823/) (2024)
- [Integrated Multi-Level Knowledge Distillation for Enhanced Speaker Verification](https://mlsanity.com/paper/2409.09389/) (2024)
- [Integrating Knowledge Distillation Methods: A Sequential Multi-Stage Framework](https://mlsanity.com/paper/2601.15657/) (2026)
- [Latent Instruction Representation Alignment: defending against jailbreaks, backdoors and undesired knowledge in LLMs](https://mlsanity.com/paper/2604.10403/) (2026)
- [Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search](https://mlsanity.com/paper/2605.21322/) (2026)
- [Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge](https://mlsanity.com/paper/2606.27527/) (2026)
- [A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions](https://mlsanity.com/paper/2607.08776/) (2026)
- [What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers](https://mlsanity.com/paper/2608.18399/) (2026)
- [Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it](https://mlsanity.com/paper/2608.23752/) (2026)
- [Can Scene Text Recognition Read Rare Compositions?](https://mlsanity.com/paper/2609.00816/) (2026)
- [Unknown-Traffic Detection, Calibration and Shortcut Reliance in Distilled Encrypted-Traffic Classifiers over One Year](https://mlsanity.com/paper/2609.31141/) (2026)
- [Know Thyself, Teach Thyself: Internal Information Flow for Selective Self-Distillation](https://mlsanity.com/paper/2609.36695/) (2026)

## Abstract (arXiv)

Knowledge distillation aims to transfer useful information from a teacher network to a student network, with the primary goal of improving the student's performance for the task at hand. Over the years, there has a been a deluge of novel techniques and use cases of knowledge distillation. Yet, despite the various improvements, there seems to be a glaring gap in the community's fundamental understanding of the process. Specifically, what is the knowledge that gets distilled in knowledge distillation? In other words, in what ways does the student become similar to the teacher? Does it start to localize objects in the same way? Does it get fooled by the same adversarial samples? Does its data invariance properties become similar? Our work presents a comprehensive study to try to answer these questions. We show that existing methods can indeed indirectly distill these properties beyond improving task performance. We further study why knowledge distillation might work this way, and show that our findings have practical implications as well.