mlsanityData to 14 Oct 2026

Knowledge Distillation from A Stronger Teacher

arXiv 2205.10536 ↗First posted 21 May 2022cs.CV cs.AI cs.LGcited by 208

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Unlike existing knowledge distillation methods focus on the baseline settings, where the teacher models and training strategies are not that strong and competing as state-of-the-art approaches, this paper presents a method dubbed DIST to distill better from a stronger teacher. We empirically find that the discrepancy of predictions between the student and a stronger teacher may tend to be fairly severer. As a result, the exact match of predictions in KL divergence would disturb the training and make existing methods perform poorly. In this paper, we show that simply preserving the relations between the predictions of teacher and student would suffice, and propose a correlation-based loss to capture the intrinsic inter-class relations from the teacher explicitly. Besides, considering that different instances have different semantic similarities to each class, we also extend this relational match to the intra-class level. Our method is simple yet practical, and extensive experiments demonstrate that it adapts well to various architectures, model sizes and training strategies, and can achieve state-of-the-art performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at: https://github.com/hunto/DIST_KD .

Key facts. Cited by 208 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 40 of its 58 references that are in the map. First posted on arXiv 21 May 2022.

Builds on

40 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Distilling Knowledge via Knowledge Review19 Apr
  2. 2021Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study1 Apr

Other references · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  9. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  10. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  11. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  12. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
26 more
  1. 2016Pyramid Scene Parsing Networkcited by 2,479
  2. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  3. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  5. 2020Knowledge Distillation: A Surveycited by 1,603
  6. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  7. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  8. 2019Relational Knowledge Distillationcited by 889
  9. 2019Contrastive Representation Distillationcited by 748
  10. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  11. 2019Similarity-Preserving Knowledge Distillationcited by 516
  12. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 429
  13. 2019On the Efficacy of Knowledge Distillationcited by 381
  14. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  15. 2019A Comprehensive Overhaul of Feature Distillationcited by 348
  16. 2019Variational Information Distillation for Knowledge Transfercited by 324
  17. 2019Correlation Congruence for Knowledge Distillationcited by 285
  18. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  19. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  20. 2020Channel-wise Knowledge Distillation for Dense Predictioncited by 163
  21. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 118
  22. 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 83
  23. 2020GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernetcited by 69
  24. 2018Learning Student Networks via Feature Embeddingcited by 45
  25. 2022DyRep: Bootstrapping Training with Dynamic Re-parameterizationcited by 11
  26. 2022Relational Surrogate Loss Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 41; 2024: 72; 2025: 57; 2026: 36.

208 papers.

Sort:

By area:

  1. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 311
  2. 2024Logit Standardization in Knowledge Distillationcited by 102
  3. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 95
  4. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 78
  5. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
  6. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  7. 2023Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matchingcited by 69
  8. 2023Decoupled Kullback-Leibler Divergence Losscited by 58
  9. 2023Knowledge Diffusion for Distillationcited by 53
  10. 2024Knowledge Distillation Based on Transformed Teacher Matchingcited by 37
  11. 2024Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillationcited by 27
  12. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
38 more
  1. 2022Masked Distillation with Receptive Tokenscited by 23
  2. 2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergencecited by 23
  3. 2024Direct Preference Knowledge Distillation for Large Language Modelscited by 21
  4. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 20
  5. 2024BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillationcited by 19
  6. 2024Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillationcited by 19
  7. 2026Demystifying On-Policy Distillation: Roles, Pathologies, and Regulationscited by 19
  8. 2023Improving Knowledge Distillation via Regularizing Feature Norm and Directioncited by 18
  9. 2024Rethinking Centered Kernel Alignment in Knowledge Distillationcited by 17
  10. 2025Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognitioncited by 16
  11. 2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 16
  12. 2024Data-efficient Large Vision Models through Sequential Autoregressioncited by 14
  13. 2023Knowledge Distillation in Vision Transformers: A Critical Reviewcited by 13
  14. 2023Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labelscited by 12
  15. 2023Exploring the Application of Large-scale Pre-trained Models on Adverse Weather Removalcited by 12
  16. 2024Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learningcited by 12
  17. 2024TelME: Teacher-leading Multimodal Fusion Network for Emotion Recognition in Conversationcited by 12
  18. 2023Student-friendly Knowledge Distillationcited by 11
  19. 2023VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scalecited by 11
  20. 2024ScaleKD: Strong Vision Transformers Could Be Excellent Teacherscited by 11
  21. 2025CustomKD: Customizing Large Vision Foundation for Edge Model Improvement via Knowledge Distillationcited by 11
  22. 2023A Simple Recipe for Competitive Low-compute Self supervised Vision Modelscited by 10
  23. 2024Transferring Core Knowledge via Learngenescited by 10
  24. 2024Weakly Supervised Monocular 3D Detection with a Single-View Imagecited by 10
  25. 2023How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectivescited by 9
  26. 2023Distilling Out-of-Distribution Robustness from Vision-Language Foundation Modelscited by 9
  27. 2024Label-efficient Semantic Scene Completion with Scribble Annotationscited by 9
  28. 2024D2LLM: Decomposed and Distilled Large Language Models for Semantic Searchcited by 9
  29. 2023Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillationcited by 8
  30. 2023Are Large Kernels Better Teachers than Transformers for ConvNets?cited by 8
  31. 2024Direct Distillation between Different Domainscited by 8
  32. 2024LIX: Implicitly Infusing Spatial Geometric Prior Knowledge into Visual Semantic Segmentation for Autonomous Drivingcited by 8
  33. 2023Avatar Knowledge Distillation: Self-ensemble Teacher Paradigm with Uncertaintycited by 7
  34. 2023Positive Label Is All You Need for Multi-Label Classificationcited by 7
  35. 2024CKD: Contrastive Knowledge Distillation from A Sample-wise Perspectivecited by 7
  36. 2024Histo-Genomic Knowledge Distillation For Cancer Prognosis From Histopathology Whole Slide Imagescited by 6
  37. 2024DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learningcited by 6
  38. 2024Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networkscited by 6

50 most cited of 208 All 208 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Knowledge Distillation with the Reused Teacher Classifier
  2. 2021Fixing the Teacher-Student Knowledge Discrepancy in Distillation
  3. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation
  4. 2022Knowledge Distillation Meets Open-Set Semi-Supervised Learning
  5. 2021Distilling Knowledge via Intermediate Classifiers
  6. 2024Multi-Level Feature Distillation of Joint Teachers Trained on Distinct Image Datasets
  7. 2024Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentation
  8. 2021Teacher's pet: understanding and mitigating biases in distillation