mlsanityData to 7 Oct 2026

DeiT III: Revenge of the ViT

arXiv 2204.07118 ↗First posted 14 Apr 2022cs.CVcited by 417

Not on the map. Why?

Abstract (arXiv)

A Vision Transformer (ViT) is a simple neural architecture amenable to serve several computer vision tasks. It has limited built-in architectural priors, in contrast to more recent architectures that incorporate priors either about the input data or of specific tasks. Recent works show that ViTs benefit from self-supervised pre-training, in particular BerT-like pre-training like BeiT. In this paper, we revisit the supervised training of ViTs. Our procedure builds upon and simplifies a recipe introduced for training ResNet-50. It includes a new simple data-augmentation procedure with only 3 augmentations, closer to the practice in self-supervised learning. Our evaluations on Image classification (ImageNet-1k with and without pre-training on ImageNet-21k), transfer learning and semantic segmentation show that our procedure outperforms by a large margin previous fully supervised training recipes for ViT. It also reveals that the performance of our ViT trained with supervision is comparable to that of more recent architectures. Our results could serve as better baselines for recent self-supervised approaches demonstrated on ViT.

Key facts. Cited by 417 papers in the map (data through 7 Oct 2026). Builds on 56 of its 68 references that are in the map. First posted on arXiv 14 Apr 2022.

Builds on

56 of 68 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2014Going Deeper with Convolutionscited by 4,231
  2. 2022A ConvNet for the 2020scited by 3,545
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  7. 2017Decoupled Weight Decay Regularizationcited by 1,722
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  9. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  10. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  11. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  12. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  13. 2016Deep Networks with Stochastic Depthcited by 973
  14. 2020Designing Network Design Spacescited by 954
  15. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  16. 2021Multiscale Vision Transformerscited by 896
  17. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  18. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  19. 2021Going deeper with Image Transformerscited by 759
  20. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  21. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  22. 2021Transformer in Transformercited by 632
  23. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  24. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  25. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  26. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  27. 2021Early Convolutions Help Transformers See Bettercited by 395
  28. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  29. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  30. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  31. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  32. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  33. 2021Video Transformer Networkcited by 271
  34. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 225
  35. 2019Fixing the train-test resolution discrepancycited by 209
  36. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  37. 2020Feature Space Augmentation for Long-Tailed Datacited by 136
  38. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 41; 2023: 100; 2024: 113; 2025: 81; 2026: 82.

417 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2023Vision Transformers Need Registerscited by 844
  4. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  5. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  6. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  7. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 395
  8. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  9. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  10. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  11. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  12. 2023A Cookbook of Self-Supervised Learningcited by 248
38 more
  1. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  2. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 235
  3. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  4. 2024Rotary Position Embedding for Vision Transformercited by 226
  5. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  6. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  7. 2023MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimationcited by 173
  8. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  9. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  10. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  11. 2022FlexiViT: One Model for All Patch Sizescited by 151
  12. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  13. 2024EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mambacited by 146
  14. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  15. 2023In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluationcited by 132
  16. 2023Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networkscited by 129
  17. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  18. 20234M: Massively Multimodal Masked Modelingcited by 118
  19. 2022Better plain ViT baselines for ImageNet-1kcited by 97
  20. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 96
  21. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 87
  22. 2023Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Modelscited by 79
  23. 2024Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performancecited by 70
  24. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 70
  25. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  26. 2023The effectiveness of MAE pre-pretraining for billion-scale pretrainingcited by 68
  27. 2023FasterViT: Fast Vision Transformers with Hierarchical Attentioncited by 65
  28. 2023UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognitioncited by 64
  29. 2024Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Modelcited by 62
  30. 2023No Representation Rules Them All in Category Discoverycited by 61
  31. 2025Your ViT is Secretly an Image Segmentation Modelcited by 61
  32. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  33. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  34. 2024Reconstructive Visual Instruction Tuningcited by 54
  35. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  36. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 51
  37. 2024Vision-LSTM: xLSTM as Generic Vision Backbonecited by 51
  38. 2023Data-Efficient Image Quality Assessment with Attention-Panel Decodercited by 49

50 most cited of 417 All 417 (JSON)