mlsanityData to 14 Oct 2026

Training data-efficient image transformers & distillation through attention

arXiv 2012.12877 ↗First posted 23 Dec 2020cs.CVcited by 4,586

Not on the map. Why?

Cited by 155 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. However, these visual transformers are pre-trained with hundreds of millions of images using an expensive infrastructure, thereby limiting their adoption. In this work, we produce a competitive convolution-free transformer by training on Imagenet only. We train them on a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external data. More importantly, we introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention. We show the interest of this token-based distillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets for both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and models.

Key facts. Cited by 4,586 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 526; 2022: 944; 2023: 989; 2024: 888; 2025: 728; 2026: 511.

4,586 papers.

Sort:

By area:

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2022A ConvNet for the 2020scited by 3,579
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  9. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,887
  10. 2021Vision Transformers for Dense Predictioncited by 1,803
  11. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,627
  12. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
38 more
  1. 2021ViViT: A Video Vision Transformercited by 1,620
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  4. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  5. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  6. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  8. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  9. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,222
  10. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  11. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  12. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  13. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  14. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  15. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  16. 2021Video Swin Transformercited by 1,091
  17. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  18. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  19. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  20. 2022Token Merging: Your ViT But Fastercited by 1,021
  21. 2021Transformers in Vision: A Surveycited by 1,013
  22. 2021Scaling Vision Transformerscited by 999
  23. 2024VMamba: Visual State Space Modelcited by 969
  24. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 959
  25. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  26. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  27. 2021Multiscale Vision Transformerscited by 897
  28. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  29. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  30. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  31. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  32. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  33. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  34. 2021Going deeper with Image Transformerscited by 764
  35. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  36. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  37. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  38. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653

50 most cited of 4,586 All 4,586 (JSON)