mlsanityData to 7 Oct 2026

The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformers

arXiv 2302.10494 ↗First posted 21 Feb 2023cs.LG cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation is an effective method for training lightweight vision models. However, acquiring teacher supervision for training samples is often costly, especially from large-scale models like vision transformers (ViTs). In this paper, we develop a simple framework to reduce the supervision cost of ViT distillation: masking out a fraction of input tokens given to the teacher. By masking input tokens, one can skip the computations associated with the masked tokens without requiring any change to teacher parameters or architecture. We find that masking patches with the lowest student attention scores is highly effective, saving up to 50% of teacher FLOPs without any drop in student accuracy, while other masking criterion leads to suboptimal efficiency gains. Through in-depth analyses, we reveal that the student-guided masking provides a good curriculum to the student, making teacher supervision easier to follow during the early stage and challenging in the later stage.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 41 of its 62 references that are in the map. First posted on arXiv 21 Feb 2023.

Builds on

41 of 62 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2022Scalable Diffusion Models with Transformerscited by 5,931
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2022Token Merging: Your ViT But Fastercited by 995
29 more
  1. 2021Scaling Vision Transformerscited by 993
  2. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  3. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  4. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  5. 2021Going deeper with Image Transformerscited by 759
  6. 2021AST: Audio Spectrogram Transformercited by 634
  7. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  8. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  9. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  10. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  11. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  12. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  13. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  14. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  15. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  16. 2021You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detectioncited by 170
  17. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 117
  18. 2022Curriculum Temperature for Knowledge Distillationcited by 110
  19. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
  20. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  21. 2022Unified Visual Transformer Compressioncited by 79
  22. 2019Knowledge Distillation via Route Constrained Optimizationcited by 73
  23. 2021Learning Efficient Vision Transformers via Fine-Grained Manifold Distillationcited by 60
  24. 2022SdAE: Self-distillated Masked Autoencodercited by 52
  25. 2021A Fast Knowledge Distillation Framework for Visual Recognitioncited by 50
  26. 2021ViDT: An Efficient and Effective Fully Transformer-based Object Detectorcited by 45
  27. 2022Optimizing Relevance Maps of Vision Transformers Improves Robustnesscited by 41
  28. 2022A Unified View of Masked Image Modelingcited by 36
  29. 2022Attention Distillation: self-supervised vision transformer students need more guidancecited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 5; 2026: 1.

8 papers.

  1. 2025Ranked Entropy Minimization for Continual Test-Time Adaptationcited by 17
  2. 2023CM-MaskSD: Cross-Modality Masked Self-Distillation for Referring Image Segmentationcited by 15
  3. 2024Vision Transformer-based Semantic Communications With Importance-Aware Quantizationcited by 5
  4. 2025Distillation Dynamics: Towards Understanding Feature-Based Distillation in Vision Transformerscited by 2
  5. 2025ReGATE: Learning Faster and Better with Fewer Tokens in MLLMscited by 1
  6. 2025CoMAD: A Multiple-Teacher Self-Supervised Distillation Frameworkcited by 0
  7. 2025Dynamic Pattern Alignment Learning for Pretraining Lightweight Human-Centric Vision Modelscited by 0
  8. 2026FTerViT: Fully Ternary Vision Transformercited by 0