mlsanityData to 7 Oct 2026

Scaling Vision Transformers

arXiv 2106.04560 ↗First posted 8 Jun 2021cs.CV cs.AI cs.LGcited by 993

Not on the map. Why?

Cited by 46 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Attention-based neural networks such as the Vision Transformer (ViT) have recently attained state-of-the-art results on many computer vision benchmarks. Scale is a primary ingredient in attaining excellent results, therefore, understanding a model's scaling properties is a key to designing future generations effectively. While the laws for scaling Transformer language models have been studied, it is unknown how Vision Transformers scale. To address this, we scale ViT models and data, both up and down, and characterize the relationships between error rate, data, and compute. Along the way, we refine the architecture and training of ViT, reducing memory consumption and increasing accuracy of the resulting models. As a result, we successfully train a ViT model with two billion parameters, which attains a new state-of-the-art on ImageNet of 90.45% top-1 accuracy. The model also performs well for few-shot transfer, for example, reaching 84.86% top-1 accuracy on ImageNet with only 10 examples per class.

Key facts. Cited by 993 papers in the map (data through 7 Oct 2026). Builds on 40 of its 54 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

40 of 54 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
28 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  5. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  6. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  7. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  8. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  9. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  10. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  11. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  12. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  13. 2021Going deeper with Image Transformerscited by 759
  14. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  15. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  16. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  17. 2020Meta Pseudo Labelscited by 372
  18. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  19. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  20. 2020Are we done with ImageNet?cited by 329
  21. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  22. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  23. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  24. 2019Fixing the train-test resolution discrepancycited by 209
  25. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  26. 2018Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networkscited by 129
  27. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 74
  28. 2021Measuring Model Biases in the Absence of Ground Truthcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 51; 2022: 168; 2023: 274; 2024: 213; 2025: 184; 2026: 103.

993 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
38 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  4. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  5. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  6. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  7. 2022Token Merging: Your ViT But Fastercited by 995
  8. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  9. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  10. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  11. 2021Robust fine-tuning of zero-shot modelscited by 785
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  13. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  14. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  15. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  16. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  17. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  18. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  19. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  20. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  21. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  22. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  23. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  24. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  25. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  26. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  27. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  28. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  29. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  30. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  31. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  32. 2023Learning Interactive Real-World Simulatorscited by 440
  33. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  34. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  35. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 418
  36. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  37. 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 389
  38. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377

50 most cited of 993 All 993 (JSON)