mlsanityData to 14 Oct 2026

Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching

arXiv 2102.02973 ↗First posted 5 Feb 2021cs.LGcited by 79

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Knowledge distillation extracts general knowledge from a pre-trained teacher network and provides guidance to a target student network. Most studies manually tie intermediate features of the teacher and student, and transfer knowledge through pre-defined links. However, manual selection often constructs ineffective links that limit the improvement from the distillation. There has been an attempt to address the problem, but it is still challenging to identify effective links under practical scenarios. In this paper, we introduce an effective and efficient feature distillation method utilizing all the feature levels of the teacher without manually selecting the links. Specifically, our method utilizes an attention-based meta-network that learns relative similarities between features, and applies identified similarities to control distillation intensities of all possible pairs. As a result, our method determines competent links more efficiently than the previous approach and provides better performance on model compression and transfer learning tasks. Further qualitative analyses and ablative studies describe how our method contributes to better distillation. The implementation code is available at github.com/clovaai/attention-feature-distillation.

Key facts. Cited by 79 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 18 of its 31 references that are in the map. First posted on arXiv 5 Feb 2021.

Builds on

18 of 31 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Wide Residual Networkscited by 3,070
  5. 2016Learning without Forgettingcited by 2,496
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  7. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  8. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  9. 2019Relational Knowledge Distillationcited by 889
  10. 2019Contrastive Representation Distillationcited by 748
  11. 2019Similarity-Preserving Knowledge Distillationcited by 516
  12. 2019Variational Information Distillation for Knowledge Transfercited by 324
6 more
  1. 2019Correlation Congruence for Knowledge Distillationcited by 285
  2. 2018Bilinear Attention Networkscited by 280
  3. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  4. 2019Learning What and Where to Transfercited by 43
  5. 2017Large-Scale Domain Adaptation via Teacher-Student Learningcited by 31
  6. 2019Knowledge distillation for semi-supervised domain adaptationcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 16; 2023: 24; 2024: 12; 2025: 17; 2026: 4.

79 papers.

Sort:

By area:

  1. 2022Knowledge Distillation via the Target-aware Transformercited by 70
  2. 2025Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecastingcited by 55
  3. 2022Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillationcited by 44
  4. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  5. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  6. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  7. 2021Student Network Learning via Evolutionary Knowledge Distillationcited by 30
  8. 2022Improved Feature Distillation via Projector Ensemblecited by 27
  9. 2024Frequency Attention for Knowledge Distillationcited by 24
  10. 2023Function-Consistent Feature Distillationcited by 21
  11. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 20
  12. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 19
38 more
  1. 2023Coaching a Teachable Studentcited by 19
  2. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  3. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  4. 2022Guided Hybrid Quantization for Object detection in Multimodal Remote Sensing Imagery via One-to-one Self-teachingcited by 12
  5. 2023Semantic Scene Completion with Cleaner Selfcited by 12
  6. 2023Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcementcited by 11
  7. 2023Student-friendly Knowledge Distillationcited by 11
  8. 2025CustomKD: Customizing Large Vision Foundation for Edge Model Improvement via Knowledge Distillationcited by 11
  9. 2024A Comprehensive Review of Knowledge Distillation in Computer Visioncited by 10
  10. 2023Cross Architecture Distillation for Face Recognitioncited by 9
  11. 2024Closed-Loop Unsupervised Representation Disentanglement with β-VAE Distillation and Diffusion Probabilistic Feedbackcited by 9
  12. 2023DistilVPR: Cross-Modal Knowledge Distillation for Visual Place Recognitioncited by 8
  13. 2022DistPro: Searching A Fast Knowledge Distillation Process via Meta Optimizationcited by 7
  14. 2024SAM-Lightening: A Lightweight Segment Anything Model with Dilated Flash Attention to Achieve 30 times Accelerationcited by 7
  15. 2024Transferring Knowledge from Large Foundation Models to Small Downstream Modelscited by 7
  16. 2023Indirect Gradient Matching for Adversarial Robust Distillationcited by 6
  17. 2024Can Distillation Mitigate Backdoor Attacks in Pre-trained Encoders?cited by 6
  18. 2025AugFL: Augmenting Federated Learning with Pretrained Modelscited by 6
  19. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  20. 2022Bridging Fairness and Environmental Sustainability in Natural Language Processingcited by 5
  21. 2023Towards Efficient Task-Driven Model Reprogramming with Foundation Modelscited by 5
  22. 2025Perspective-Aware Teaching: Adapting Knowledge for Heterogeneous Distillationcited by 5
  23. 2025Uncertainty-Aware Knowledge Distillation for Compact and Efficient 6DoF Pose Estimationcited by 5
  24. 2025Mitigating Semantic Collapse in Partially Relevant Video Retrievalcited by 5
  25. 2023ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppressioncited by 4
  26. 2023Leveraging Angular Distributions for Improved Knowledge Distillationcited by 3
  27. 2022Auto-Transfer: Learning to Route Transferrable Representationscited by 2
  28. 2022Knowledge Distillation with Deep Supervisioncited by 2
  29. 2022Variance Tolerance Factors For Interpreting ALL Neural Networkscited by 2
  30. 2022Hilbert Distillation for Cross-Dimensionality Networkscited by 2
  31. 2023Filter Pruning for Efficient CNNs via Knowledge-driven Differential Filter Samplercited by 2
  32. 2023Rethinking Intermediate Layers design in Knowledge Distillation for Kidney and Liver Tumor Segmentationcited by 2
  33. 2023Dual-Teacher De-biasing Distillation Framework for Multi-domain Fake News Detectioncited by 2
  34. 2024From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networkscited by 2
  35. 2025Hybrid Knowledge Transfer through Attention and Logit Distillation for On-Device Vision Systems in Agricultural IoTcited by 2
  36. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  37. 2025Selective Structured State Space for Multispectral-fused Small Target Detectioncited by 2
  38. 2025Distillation Dynamics: Towards Understanding Feature-Based Distillation in Vision Transformerscited by 2

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Fixing the Teacher-Student Knowledge Discrepancy in Distillation
  2. 2021Distilling Knowledge via Knowledge Review
  3. 2026A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs
  4. 2021Knowledge Distillation via Instance-level Sequence Learning
  5. 2024Feature Alignment-Based Knowledge Distillation for Efficient Compression of Large Language Models
  6. 2025Contrastive Representation Distillation via Multi-Scale Feature Decoupling
  7. 2022On effects of Knowledge Distillation on Transfer Learning
  8. 2021Does Knowledge Distillation Really Work?