mlsanityData to 14 Oct 2026

Class Attention Transfer Based Knowledge Distillation

arXiv 2304.12777 ↗First posted 25 Apr 2023cs.CVcited by 61

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Previous knowledge distillation methods have shown their impressive performance on model compression tasks, however, it is hard to explain how the knowledge they transferred helps to improve the performance of the student network. In this work, we focus on proposing a knowledge distillation method that has both high interpretability and competitive performance. We first revisit the structure of mainstream CNN models and reveal that possessing the capacity of identifying class discriminative regions of input is critical for CNN to perform classification. Furthermore, we demonstrate that this capacity can be obtained and enhanced by transferring class activation maps. Based on our findings, we propose class attention transfer based knowledge distillation (CAT-KD). Different from previous KD methods, we explore and present several properties of the knowledge transferred by our method, which not only improve the interpretability of CAT-KD but also contribute to a better understanding of CNN. While having high interpretability, CAT-KD achieves state-of-the-art performance on multiple benchmarks. Code is available at: https://github.com/GzyAftermath/CAT-KD.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 30 of its 38 references that are in the map. First posted on arXiv 25 Apr 2023.

Builds on

30 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Decoupled Knowledge Distillation16 Mar
  2. 2021Distilling Knowledge via Knowledge Review19 Apr

Other references · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2016Densely Connected Convolutional Networkscited by 6,720
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2016Wide Residual Networkscited by 3,070
  8. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  10. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  11. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  12. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
16 more
  1. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 939
  2. 2019Relational Knowledge Distillationcited by 889
  3. 2019Contrastive Representation Distillationcited by 748
  4. 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 582
  5. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  6. 2018Born Again Neural Networkscited by 546
  7. 2019Similarity-Preserving Knowledge Distillationcited by 516
  8. 2019On the Efficacy of Knowledge Distillationcited by 381
  9. 2019A Comprehensive Overhaul of Feature Distillationcited by 348
  10. 2019Correlation Congruence for Knowledge Distillationcited by 285
  11. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 227
  12. 2018Tell Me Where to Look: Guided Attention Inference Networkcited by 169
  13. 2016Deep Pyramidal Residual Networkscited by 163
  14. 2018Zigzag Learning for Weakly Supervised Object Detectioncited by 24
  15. 2019Combinational Class Activation Maps for Weakly Supervised Object Localizationcited by 24
  16. 2021CAM-loss: Towards Learning Spatially Discriminative Feature Representationscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 20; 2025: 21; 2026: 17.

61 papers.

Sort:

By area:

  1. 2024Logit Standardization in Knowledge Distillationcited by 102
  2. 2024Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillationcited by 19
  3. 2024Good Teachers Explain: Explanation-Enhanced Knowledge Distillationcited by 14
  4. 2024A Comprehensive Review of Knowledge Distillation in Computer Visioncited by 10
  5. 2024Knowledge Distillation with Refined Logitscited by 9
  6. 2024Direct Distillation between Different Domainscited by 8
  7. 2024CKD: Contrastive Knowledge Distillation from A Sample-wise Perspectivecited by 7
  8. 2024DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learningcited by 6
  9. 2025VRM: Knowledge Distillation via Virtual Relation Matchingcited by 6
  10. 2024Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architecturescited by 5
  11. 2024Disentangling spatio-temporal knowledge for weakly supervised object detection and segmentation in surgical videocited by 5
  12. 2025DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainercited by 4
38 more
  1. 2025Local Dense Logit Relations for Enhanced Knowledge Distillationcited by 4
  2. 2025Cross-Architecture Distillation Made Simple with Redundancy Suppressioncited by 4
  3. 2023Maximizing Discrimination Capability of Knowledge Distillation with Energy Functioncited by 3
  4. 2024Instance Temperature Knowledge Distillationcited by 3
  5. 2024Enhancing Logits Distillation with Plug&Play Kendall's τ Ranking Losscited by 3
  6. 2025HDC: Hierarchical Distillation for Multi-level Noisy Consistency in Semi-Supervised Fetal Ultrasound Segmentationcited by 3
  7. 2024Learn from Balance: Rectifying Knowledge Transfer for Long-Tailed Scenarioscited by 2
  8. 2025Multi-aspect Knowledge Distillation with Large Language Modelcited by 2
  9. 2025Adaptive Temperature Based on Logits Correlation in Knowledge Distillationcited by 2
  10. 2025CAE-DFKD: Bridging the Transferability Gap in Data-Free Knowledge Distillationcited by 2
  11. 2025Dual-Forward Path Teacher Knowledge Distillation: Bridging the Capacity Gap Between Teacher and Studentcited by 2
  12. 2026Learning to Reason: Temporal Saliency Distillation for Interpretable Knowledge Transfercited by 2
  13. 2026Distilling Lightweight Domain Experts from Large ML Models by Identifying Relevant Subspacescited by 2
  14. 2023Review helps learn better: Temporal Supervised Knowledge Distillationcited by 1
  15. 2024Online Robot Navigation and Manipulation with Distilled Vision-Language Modelscited by 1
  16. 2024Multi-perspective Contrastive Logit Distillationcited by 1
  17. 2024Diffusion-based Data Augmentation and Knowledge Distillation with Generated Soft Labels Solving Data Scarcity Problems of SAR Oil Spill Segmentationcited by 1
  18. 2025Task-Specific Knowledge Distillation from the Vision Foundation Model for Enhanced Medical Image Segmentationcited by 1
  19. 2025Swapped Logit Distillation via Bi-level Teacher Alignmentcited by 1
  20. 2025DCSNet: A Lightweight Knowledge Distillation-Based Model with Explainable AI for Lung Cancer Diagnosis from Histopathological Imagescited by 1
  21. 2025Structure-Aware Automatic Channel Pruning by Searching with Graph Embeddingcited by 1
  22. 2025Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversitycited by 1
  23. 2026Beyond Student: An Asymmetric Network for Neural Network Inheritancecited by 1
  24. 2023On Preserving the Knowledge of Long Clinical Textscited by 0
  25. 2024Estimating Human Poses Across Datasets: A Unified Skeleton and Multi-Teacher Distillation Approachcited by 0
  26. 2024Discriminative and Consistent Representation Distillationcited by 0
  27. 2024Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teachercited by 0
  28. 2024Dynamic Textual Prompt For Rehearsal-free Lifelong Person Re-identificationcited by 0
  29. 2025Rethinking Knowledge in Distillation: An In-context Sample Retrieval Perspectivecited by 0
  30. 2025Contrastive Representation Distillation via Multi-Scale Feature Decouplingcited by 0
  31. 2025Multi-Level Decoupled Relational Distillation for Heterogeneous Architecturescited by 0
  32. 2025FiGKD: Fine-Grained Knowledge Distillation via High-Frequency Detail Transfercited by 0
  33. 2025Distilling Balanced Knowledge from a Biased Teachercited by 0
  34. 2025TopKD: Top-scaled Knowledge Distillationcited by 0
  35. 2025Enriching Knowledge Distillation with Intra-Class Contrastive Learningcited by 0
  36. 2026Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Modelcited by 0
  37. 2026REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiencycited by 0
  38. 2026Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentationcited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Attention-based Knowledge Distillation in Multi-attention Tasks: The Impact of a DCT-driven Loss
  2. 2022On effects of Knowledge Distillation on Transfer Learning
  3. 2023Improving Knowledge Distillation with Teacher's Explanation
  4. 2025Knowledge Distillation: Enhancing Neural Network Compression with Integrated Gradients
  5. 2025A Comprehensive Survey on Knowledge Distillation
  6. 2021Compact CNN Structure Learning by Knowledge Distillation
  7. 2024Frequency Attention for Knowledge Distillation
  8. 2022Towards a Unified View of Affinity-Based Knowledge Distillation