mlsanityData to 14 Oct 2026

Knowledge Distillation via the Target-aware Transformer

arXiv 2205.10793 ↗First posted 22 May 2022cs.CVcited by 70

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Knowledge distillation becomes a de facto standard to improve the performance of small neural networks. Most of the previous works propose to regress the representational features from the teacher to the student in a one-to-one spatial matching fashion. However, people tend to overlook the fact that, due to the architecture differences, the semantic information on the same spatial location usually vary. This greatly undermines the underlying assumption of the one-to-one distillation approach. To this end, we propose a novel one-to-all spatial matching knowledge distillation approach. Specifically, we allow each pixel of the teacher feature to be distilled to all spatial locations of the student features given its similarity, which is generated from a target-aware transformer. Our approach surpasses the state-of-the-art methods by a significant margin on various computer vision benchmarks, such as ImageNet, Pascal VOC and COCOStuff10k. Code is available at https://github.com/sihaoevery/TaT.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 36 of its 55 references that are in the map. First posted on arXiv 22 May 2022.

Builds on

36 of 55 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillation8 Feb
  2. 2021Distilling Knowledge via Knowledge Review19 Apr
  3. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching5 Feb

Other references · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  10. 2014Going Deeper with Convolutionscited by 4,241
  11. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  12. 2016Wide Residual Networkscited by 3,070
21 more
  1. 2016Learning without Forgettingcited by 2,496
  2. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  3. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  5. 2017Deformable Convolutional Networkscited by 1,488
  6. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  7. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  8. 2019Relational Knowledge Distillationcited by 889
  9. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  10. 2019Contrastive Representation Distillationcited by 748
  11. 2019Similarity-Preserving Knowledge Distillationcited by 516
  12. 2013Do Deep Nets Really Need to be Deep?cited by 474
  13. 2019A Comprehensive Overhaul of Feature Distillationcited by 348
  14. 2019Correlation Congruence for Knowledge Distillationcited by 285
  15. 2019Structured Knowledge Distillation for Dense Predictioncited by 279
  16. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  17. 2020Cross-Layer Distillation with Semantic Calibrationcited by 143
  18. 2021General Instance Distillation for Object Detectioncited by 114
  19. 2020Spatio-Temporal Graph for Video Captioning with Knowledge Distillationcited by 109
  20. 2019Knowledge Adaptation for Efficient Semantic Segmentationcited by 105
  21. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 101

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 24; 2024: 21; 2025: 13; 2026: 7.

70 papers.

Sort:

By area:

  1. 2024Logit Standardization in Knowledge Distillationcited by 102
  2. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  3. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
  4. 2022In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimationcited by 51
  5. 2022Cross-Architecture Knowledge Distillationcited by 41
  6. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  7. 2022ViTKD: Practical Guidelines for ViT feature knowledge distillationcited by 34
  8. 2023Weight subcloning: direct initialization of transformers using larger pretrained onescited by 30
  9. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  10. 2023Mask Propagation for Efficient Video Semantic Segmentationcited by 26
  11. 2022Cross-modal Clinical Graph Transformer for Ophthalmic Report Generationcited by 24
  12. 2023Function-Consistent Feature Distillationcited by 21
38 more
  1. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 20
  2. 2023DomainAdaptor: A Novel Approach to Test-time Adaptationcited by 20
  3. 2024MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolutioncited by 17
  4. 2023Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillationcited by 15
  5. 2023Knowledge Distillation in Vision Transformers: A Critical Reviewcited by 13
  6. 2023NormKD: Normalized Logits for Knowledge Distillationcited by 12
  7. 2023VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scalecited by 11
  8. 2023Soft Prompt Decoding for Multilingual Dense Retrievalcited by 10
  9. 2025Mix-QViT: Mixed-Precision Vision Transformer Quantization Driven by Layer Importance and Quantization Sensitivitycited by 9
  10. 2024LRP-QViT: Mixed-Precision Vision Transformer Quantization via Layer-wise Relevance Propagationcited by 8
  11. 2023BinaryViT: Towards Efficient and Accurate Binary Vision Transformerscited by 7
  12. 2023DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Modelscited by 7
  13. 2023Generative Model-based Feature Knowledge Distillation for Action Recognitioncited by 7
  14. 2024CKD: Contrastive Knowledge Distillation from A Sample-wise Perspectivecited by 7
  15. 2023A Simple and Generic Framework for Feature Distillation via Channel-wise Transformationcited by 6
  16. 2023Video Recognition in Portrait Modecited by 6
  17. 2024Dynamic Contrastive Knowledge Distillation for Efficient Image Restorationcited by 6
  18. 2025VRM: Knowledge Distillation via Virtual Relation Matchingcited by 6
  19. 2024Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architecturescited by 5
  20. 2025Random Conditioning with Distillation for Data-Efficient Diffusion Model Compressioncited by 5
  21. 2023Influence Function Based Second-Order Channel Pruning-Evaluating True Loss Changes For Pruning Is Possible Without Retrainingcited by 4
  22. 2024Deep video representation learning: a surveycited by 4
  23. 2025Mix-QSAM: Mixed-Precision Quantization of the Segment Anything Modelcited by 4
  24. 2025Cross-Architecture Distillation Made Simple with Redundancy Suppressioncited by 4
  25. 2023RdimKD: Generic Distillation Paradigm by Dimensionality Reductioncited by 3
  26. 2024LoReTrack: Efficient and Accurate Low-Resolution Transformer Trackingcited by 3
  27. 2024From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networkscited by 2
  28. 2025Delving Deep into Semantic Relation Distillationcited by 2
  29. 2022Online Cross-Layer Knowledge Distillation on Graph Neural Networks with Deep Supervisioncited by 1
  30. 2023Channel-Aware Distillation Transformer for Depth Estimation on Nano Dronescited by 1
  31. 2024Ranking Distillation for Open-Ended Video Question Answering with Insufficient Labelscited by 1
  32. 2024An approach to optimize inference of the DIART speaker diarization pipelinecited by 1
  33. 2024Computer Vision Model Compression Techniques for Embedded Systems: A Surveycited by 1
  34. 2024Harmonizing knowledge Transfer in Neural Network with Unified Distillationcited by 1
  35. 2024CLFace: A Scalable and Resource-Efficient Continual Learning Framework for Lifelong Face Recognitioncited by 1
  36. 2025Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversitycited by 1
  37. 2025UHKD: A Unified Framework for Heterogeneous Knowledge Distillation via Frequency-Domain Representationscited by 1
  38. 2026Beyond Student: An Asymmetric Network for Neural Network Inheritancecited by 1

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023NORM: Knowledge Distillation via N-to-One Representation Matching
  2. 2023Cumulative Spatial Knowledge Distillation for Vision Transformers
  3. 2022Knowledge Distillation from A Stronger Teacher
  4. 2025KD2M: A unifying framework for feature knowledge distillation
  5. 2024Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers
  6. 2021Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition
  7. 2022Attention-based Knowledge Distillation in Multi-attention Tasks: The Impact of a DCT-driven Loss
  8. 2021Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation