mlsanityData to 7 Oct 2026

What Knowledge Gets Distilled in Knowledge Distillation?

arXiv 2205.16004 ↗First posted 31 May 2022cs.CV cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation aims to transfer useful information from a teacher network to a student network, with the primary goal of improving the student's performance for the task at hand. Over the years, there has a been a deluge of novel techniques and use cases of knowledge distillation. Yet, despite the various improvements, there seems to be a glaring gap in the community's fundamental understanding of the process. Specifically, what is the knowledge that gets distilled in knowledge distillation? In other words, in what ways does the student become similar to the teacher? Does it start to localize objects in the same way? Does it get fooled by the same adversarial samples? Does its data invariance properties become similar? Our work presents a comprehensive study to try to answer these questions. We show that existing methods can indeed indirectly distill these properties beyond improving task performance. We further study why knowledge distillation might work this way, and show that our findings have practical implications as well.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 31 May 2022.

Builds on

32 of 43 references.

References · 32

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  10. 2020Knowledge Distillation: A Surveycited by 1,578
  11. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  12. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
20 more
  1. 2019When Does Label Smoothing Help?cited by 985
  2. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  3. 2016Adversarial Machine Learning at Scalecited by 934
  4. 2019Relational Knowledge Distillationcited by 873
  5. 2019Contrastive Representation Distillationcited by 734
  6. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  7. 2017Deep Mutual Learningcited by 621
  8. 2018Born Again Neural Networkscited by 539
  9. 2019Similarity-Preserving Knowledge Distillationcited by 507
  10. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  11. 2019On the Efficacy of Knowledge Distillationcited by 377
  12. 2019Correlation Congruence for Knowledge Distillationcited by 280
  13. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  14. 2021Does Knowledge Distillation Really Work?cited by 187
  15. 2021Partial success in closing the gap between human and machine visioncited by 185
  16. 2021Towards Understanding Knowledge Distillationcited by 158
  17. 2019Adversarially Robust Distillationcited by 124
  18. 2020Self-Distillation as Instance-Specific Label Smoothingcited by 71
  19. 2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
  20. 2021Teacher's pet: understanding and mitigating biases in distillationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 5; 2026: 12.

26 papers.

Sort:

By area:

  1. 2024Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillationcited by 18
  2. 2023On student-teacher deviations in distillation: does it pay to disobey?cited by 16
  3. 2024Provable Weak-to-Strong Generalization via Benign Overfittingcited by 16
  4. 2024Good Teachers Explain: Explanation-Enhanced Knowledge Distillationcited by 14
  5. 2025Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimensioncited by 10
  6. 2024CKD: Contrastive Knowledge Distillation from A Sample-wise Perspectivecited by 6
  7. 2025Random Conditioning with Distillation for Data-Efficient Diffusion Model Compressioncited by 5
  8. 2024Zero-Shot Distillation for Image Encoders: How to Make Effective Use of Synthetic Datacited by 4
  9. 2025Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applicationscited by 4
  10. 2023Group Distributionally Robust Knowledge Distillationcited by 3
  11. 2025Efficient Verified Machine Unlearning For Distillationcited by 2
  12. 2026Distilling Lightweight Domain Experts from Large ML Models by Identifying Relevant Subspacescited by 2
14 more
  1. 2025Does Weak-to-strong Generalization Happen under Spurious Correlations?cited by 1
  2. 2026Knowledge Distillation Must Account for What It Losescited by 1
  3. 2023Weight Averaging Improves Knowledge Distillation under Domain Shiftcited by 0
  4. 2024Stereo-Knowledge Distillation from dpMV to Dual Pixels for Light Field Video Reconstructioncited by 0
  5. 2026Integrating Knowledge Distillation Methods: A Sequential Multi-Stage Frameworkcited by 0
  6. 2026Latent Instruction Representation Alignment: defending against jailbreaks, backdoors and undesired knowledge in LLMscited by 0
  7. 2026Optimized Federated Knowledge Distillation with Distributed Neural Architecture Searchcited by 0
  8. 2026Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledgecited by 0
  9. 2026A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactionscited by 0
  10. 2026What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformerscited by 0
  11. 2026Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid itcited by 0
  12. 2026Can Scene Text Recognition Read Rare Compositions?cited by 0
  13. 2026Unknown-Traffic Detection, Calibration and Shortcut Reliance in Distilled Encrypted-Traffic Classifiers over One Yearcited by 0
  14. 2026Know Thyself, Teach Thyself: Internal Information Flow for Selective Self-Distillationcited by 0