mlsanityData to 7 Oct 2026

How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

arXiv 2106.10270 ↗First posted 18 Jun 2021cs.CV cs.AI cs.LGcited by 562

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViT) have been shown to attain highly competitive performance for a wide range of vision applications, such as image classification, object detection and semantic image segmentation. In comparison to convolutional neural networks, the Vision Transformer's weaker inductive bias is generally found to cause an increased reliance on model regularization or data augmentation ("AugReg" for short) when training on smaller training datasets. We conduct a systematic empirical study in order to better understand the interplay between the amount of training data, AugReg, model size and compute budget. As one result of this study we find that the combination of increased compute and AugReg can yield models with the same performance as models trained on an order of magnitude more training data: we train ViT models of various sizes on the public ImageNet-21k dataset which either match or outperform their counterparts trained on the larger, but not publicly available JFT-300M dataset.

Key facts. Cited by 562 papers in the map (data through 7 Oct 2026). Builds on 35 of its 50 references that are in the map. First posted on arXiv 18 Jun 2021.

Builds on

35 of 50 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2014Going Deeper with Convolutionscited by 4,231
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
23 more
  1. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  5. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  6. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  7. 2016Deep Networks with Stochastic Depthcited by 973
  8. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  9. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  10. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  11. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  12. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  13. 2021Transformer in Transformercited by 632
  14. 2021ImageNet-21K Pretraining for the Massescited by 607
  15. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  16. 2018Do Better ImageNet Models Transfer Better?cited by 560
  17. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  18. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  19. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  20. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  21. 2019Fixing the train-test resolution discrepancycited by 209
  22. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  23. 2021SiT: Self-supervised vIsion Transformercited by 99

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 122; 2023: 125; 2024: 127; 2025: 105; 2026: 56.

562 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2022A ConvNet for the 2020scited by 3,545
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  5. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  6. 2022Token Merging: Your ViT But Fastercited by 995
  7. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  8. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  9. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  10. 2022DeiT III: Revenge of the ViTcited by 417
  11. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  12. 2022Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Frameworkcited by 328
38 more
  1. 2022How Do Vision Transformers Work?cited by 305
  2. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  3. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  4. 2023A Cookbook of Self-Supervised Learningcited by 248
  5. 2024Rotary Position Embedding for Vision Transformercited by 226
  6. 2022Learning Video Representations from Large Language Modelscited by 204
  7. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  8. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  9. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  10. 2023MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimationcited by 173
  11. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  12. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  13. 2023BioCLIP: A Vision Foundation Model for the Tree of Lifecited by 159
  14. 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155
  15. 2022FlexiViT: One Model for All Patch Sizescited by 151
  16. 2024Continual Learning with Pre-Trained Models: A Surveycited by 151
  17. 2022Multiview Transformers for Video Recognitioncited by 148
  18. 2021The Benchmark Lotterycited by 133
  19. 2023In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluationcited by 132
  20. 2021PixMix: Dreamlike Pictures Comprehensively Improve Safety Measurescited by 131
  21. 2023Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networkscited by 129
  22. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  23. 2023Change is Hard: A Closer Look at Subpopulation Shiftcited by 119
  24. 2022Hyperbolic Vision Transformers: Combining Improvements in Metric Learningcited by 109
  25. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
  26. 2022Retrieval Augmented Classification for Long-Tail Visual Recognitioncited by 106
  27. 2022SegViT: Semantic Segmentation with Plain Vision Transformerscited by 105
  28. 2021The Efficiency Misnomercited by 98
  29. 2022Better plain ViT baselines for ImageNet-1kcited by 97
  30. 2024Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architecturescited by 97
  31. 2024Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observationcited by 97
  32. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  33. 2022Three things everyone should know about Vision Transformerscited by 90
  34. 2024Advancing Multimodal Medical Capabilities of Geminicited by 88
  35. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 87
  36. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 87
  37. 2023FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learningcited by 86
  38. 2022OpenEarthMap: A Benchmark Dataset for Global High-Resolution Land Cover Mappingcited by 80

50 most cited of 562 All 562 (JSON)