mlsanityData to 14 Oct 2026

Knowledge Distillation with the Reused Teacher Classifier

arXiv 2203.14001 ↗First posted 26 Mar 2022cs.CVcited by 141

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Knowledge distillation aims to compress a powerful yet cumbersome teacher model into a lightweight student model without much sacrifice of performance. For this purpose, various approaches have been proposed over the past few years, generally with elaborately designed knowledge representations, which in turn increase the difficulty of model development and interpretation. In contrast, we empirically show that a simple knowledge distillation technique is enough to significantly narrow down the teacher-student performance gap. We directly reuse the discriminative classifier from the pre-trained teacher model for student inference and train a student encoder through feature alignment with a single ℓ2 loss. In this way, the student model is able to achieve exactly the same performance as the teacher model provided that their extracted features are perfectly aligned. An additional projector is developed to help the student encoder match with the teacher classifier, which renders our technique applicable to various teacher and student architectures. Extensive experiments demonstrate that our technique achieves state-of-the-art results at the modest cost of compression ratio due to the added projector.

Key facts. Cited by 141 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 45 of its 61 references that are in the map. First posted on arXiv 26 Mar 2022.

Builds on

45 of 61 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Confidence-Aware Multi-Teacher Knowledge Distillation30 Dec
  2. 2021Distilling Knowledge via Knowledge Review19 Apr

Other references · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2016Densely Connected Convolutional Networkscited by 6,720
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2016Wide Residual Networkscited by 3,070
31 more
  1. 2016Learning without Forgettingcited by 2,496
  2. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  4. 2020Knowledge Distillation: A Surveycited by 1,603
  5. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  6. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  7. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  8. 2019Relational Knowledge Distillationcited by 889
  9. 2020Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptationcited by 848
  10. 2019Contrastive Representation Distillationcited by 748
  11. 2019Similarity-Preserving Knowledge Distillationcited by 516
  12. 2013Do Deep Nets Really Need to be Deep?cited by 474
  13. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  14. 2019Variational Information Distillation for Knowledge Transfercited by 324
  15. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  16. 2019Structured Knowledge Distillation for Dense Predictioncited by 279
  17. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  18. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  19. 2019Data-Free Learning of Student Networkscited by 204
  20. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 173
  21. 2020Channel-wise Knowledge Distillation for Dense Predictioncited by 163
  22. 2020Cross-Layer Distillation with Semantic Calibrationcited by 143
  23. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  24. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  25. 2019Online Knowledge Distillation with Diverse Peerscited by 128
  26. 2020A survey on domain adaptation theory: learning bounds and theoretical guaranteescited by 109
  27. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  28. 2016Hypothesis Transfer Learning via Transformation Functionscited by 26
  29. 2021Distilling Holistic Knowledge with Graph Neural Networkscited by 26
  30. 2014Fast Rates by Transferring from Auxiliary Hypothesescited by 16
  31. 2020Learning with Retrospectioncited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 41; 2024: 45; 2025: 31; 2026: 18.

141 papers.

Sort:

By area:

  1. 2024Logit Standardization in Knowledge Distillationcited by 102
  2. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  3. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Modelscited by 72
  4. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  5. 2023Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistencycited by 42
  6. 2023Leveraging Vision-Language Models for Improving Domain Generalization in Image Classificationcited by 41
  7. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  8. 2023Understanding the Role of the Projector in Knowledge Distillationcited by 31
  9. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  10. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  11. 2024Dual-Space Knowledge Distillation for Large Language Modelscited by 25
  12. 2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergencecited by 23
38 more
  1. 2023Function-Consistent Feature Distillationcited by 21
  2. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 20
  3. 2023Supervision Complexity and its Role in Knowledge Distillationcited by 19
  4. 2023Adaptive Multi-Teacher Knowledge Distillation with Meta-Learningcited by 16
  5. 2022Accelerating Diffusion Sampling with Classifier-based Feature Distillationcited by 15
  6. 2023Decomposed Cross-modal Distillation for RGB-based Temporal Action Detectioncited by 12
  7. 2023NormKD: Normalized Logits for Knowledge Distillationcited by 12
  8. 2023EmbedDistill: A Geometric Knowledge Distillation for Information Retrievalcited by 11
  9. 2023VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scalecited by 11
  10. 2023Cloud-Device Collaborative Learning for Multimodal Large Language Modelscited by 11
  11. 2024CrossGLG: LLM Guides One-shot Skeleton-based 3D Action Recognition in a Cross-level Mannercited by 11
  12. 2024ScaleKD: Strong Vision Transformers Could Be Excellent Teacherscited by 11
  13. 2023FoPro-KD: Fourier Prompted Effective Knowledge Distillation for Long-Tailed Medical Image Recognitioncited by 10
  14. 2023Diverse Target and Contribution Scheduling for Domain Generalizationcited by 10
  15. 2024Knowledge Distillation with Refined Logitscited by 9
  16. 2023Distribution Shift Matters for Knowledge Distillation with Webly Collected Imagescited by 8
  17. 2023Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Promptcited by 8
  18. 2024Direct Distillation between Different Domainscited by 8
  19. 2025A Dual-Space Framework for General Knowledge Distillation of Large Language Modelscited by 8
  20. 2025Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Frameworkcited by 8
  21. 2023Adjusting Logit in Gaussian Form for Long-Tailed Visual Recognitioncited by 7
  22. 2023Fundus-Enhanced Disease-Aware Distillation Model for Retinal Disease Classification from OCT Imagescited by 7
  23. 2023Using Early Readouts to Mediate Featural Bias in Distillationcited by 7
  24. 2024CKD: Contrastive Knowledge Distillation from A Sample-wise Perspectivecited by 7
  25. 2023Debiased Distillation by Transplanting the Last Layercited by 6
  26. 2023Robust Saliency-Aware Distillation for Few-shot Fine-grained Visual Recognitioncited by 6
  27. 2023MoMA: Momentum Contrastive Learning with Multi-head Attention-based Knowledge Distillation for Histopathology Image Analysiscited by 6
  28. 2024Dynamic Temperature Knowledge Distillationcited by 6
  29. 2025VRM: Knowledge Distillation via Virtual Relation Matchingcited by 6
  30. 2022Alignahead: Online Cross-Layer Knowledge Extraction on Graph Neural Networkscited by 5
  31. 2023Towards Efficient Task-Driven Model Reprogramming with Foundation Modelscited by 5
  32. 2023Cosine Similarity Knowledge Distillation for Individual Class Information Transfercited by 5
  33. 2024Scale Decoupled Distillationcited by 5
  34. 2024Improve Knowledge Distillation via Label Revision and Data Selectioncited by 5
  35. 2024MutDet: Mutually Optimizing Pre-training for Remote Sensing Object Detectioncited by 5
  36. 2024Adaptive Group Robust Ensemble Knowledge Distillationcited by 5
  37. 2024Cross-View Consistency Regularisation for Knowledge Distillationcited by 5
  38. 2025Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attackcited by 5

50 most cited of 141 All 141 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Fixing the Teacher-Student Knowledge Discrepancy in Distillation
  2. 2022Knowledge Distillation from A Stronger Teacher
  3. 2023Improving Knowledge Distillation via Regularizing Feature Norm and Direction
  4. 2021Distilling Knowledge via Intermediate Classifiers
  5. 2023Teacher-Student Architecture for Knowledge Distillation: A Survey
  6. 2025The Role of Teacher Calibration in Knowledge Distillation
  7. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation
  8. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillation