mlsanityData to 7 Oct 2026

Attention Distillation: self-supervised vision transformer students need more guidance

arXiv 2210.00944 ↗First posted 3 Oct 2022cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning has been widely applied to train high-quality vision transformers. Unleashing their excellent performance on memory and compute constraint devices is therefore an important research topic. However, how to distill knowledge from one self-supervised ViT to another has not yet been explored. Moreover, the existing self-supervised knowledge distillation (SSKD) methods focus on ConvNet based architectures are suboptimal for ViT knowledge distillation. In this paper, we study knowledge distillation of self-supervised vision transformers (ViT-SSKD). We show that directly distilling information from the crucial attention mechanism from teacher to student can significantly narrow the performance gap between both. In experiments on ImageNet-Subset and ImageNet-1K, we show that our method AttnDistill outperforms existing self-supervised knowledge distillation (SSKD) methods and achieves state-of-the-art k-NN accuracy compared with self-supervised learning (SSL) methods learning from scratch (with the ViT-S model). We are also the first to apply the tiny ViT-T model on self-supervised learning. Moreover, AttnDistill is independent of self-supervised learning algorithms, it can be adapted to ViT based SSL methods to improve the performance in future research. The code is here: https://github.com/wangkai930418/attndistill

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 72 of its 83 references that are in the map. First posted on arXiv 3 Oct 2022.

Builds on

72 of 83 references.

References · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
38 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  5. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  13. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  14. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  15. 2020Knowledge Distillation: A Surveycited by 1,578
  16. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  17. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  18. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  19. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  20. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  21. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  22. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  23. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  24. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  25. 2019Relational Knowledge Distillationcited by 873
  26. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  27. 2021Going deeper with Image Transformerscited by 759
  28. 2019Contrastive Representation Distillationcited by 734
  29. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  30. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  31. 2017Large Batch Training of Convolutional Networkscited by 529
  32. 2019Similarity-Preserving Knowledge Distillationcited by 507
  33. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  34. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  35. 2022DeiT III: Revenge of the ViTcited by 417
  36. 2022Decoupled Knowledge Distillationcited by 380
  37. 2019Variational Information Distillation for Knowledge Transfercited by 321
  38. 2021Distilling Knowledge via Knowledge Reviewcited by 285

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 7; 2026: 2.

16 papers.

Sort:

By area:

  1. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 19
  2. 2024On the Surprising Effectiveness of Attention Transfer for Vision Transformerscited by 16
  3. 2025X2I: Seamless Integration of Multimodal Understanding into Diffusion Transformer via Attention Distillationcited by 11
  4. 2023The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformerscited by 8
  5. 2024A Survey of the Self Supervised Learning Mechanisms for Vision Transformerscited by 8
  6. 2024Simple Unsupervised Knowledge Distillation With Space Similaritycited by 5
  7. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environmentscited by 5
  8. 2025LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervisioncited by 5
  9. 2026Attention Transfer Is Not Universally Effective for Vision Transformerscited by 3
  10. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  11. 2025Learning Task-Agnostic Representations through Multi-Teacher Distillationcited by 2
  12. 2023KS-DETR: Knowledge Sharing in Attention Learning for Detection Transformercited by 1
4 more
  1. 2024Knowledge Distillation in RNN-Attention Models for Early Prediction of Student Performancecited by 1
  2. 2025Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimizationcited by 1
  3. 2025PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Modelscited by 0
  4. 2026Copy the Same, Distill the Difference: Initializing Linear Vision Transformerscited by 0