mlsanityData to 14 Oct 2026

Scaling Vision Transformers to 22 Billion Parameters

arXiv 2302.05442 ↗First posted 10 Feb 2023cs.CV cs.AI cs.LGcited by 694

Not on the map. Why?

Cited by 49 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The scaling of Transformers has driven breakthrough capabilities for language models. At present, the largest large language models (LLMs) contain upwards of 100B parameters. Vision Transformers (ViT) have introduced the same architecture to image and video modelling, but these have not yet been successfully scaled to nearly the same degree; the largest dense ViT contains 4B parameters (Chen et al., 2022). We present a recipe for highly efficient and stable training of a 22B-parameter ViT (ViT-22B) and perform a wide variety of experiments on the resulting model. When evaluated on downstream tasks (often with a lightweight linear model on frozen features), ViT-22B demonstrates increasing performance with scale. We further observe other interesting benefits of scale, including an improved tradeoff between fairness and performance, state-of-the-art alignment to human visual perception in terms of shape/texture bias, and improved robustness. ViT-22B demonstrates the potential for "LLM-like" scaling in vision, and provides key steps towards getting there.

Key facts. Cited by 694 papers in the map (data through 14 Oct 2026). Builds on 87 of its 134 references that are in the map. First posted on arXiv 10 Feb 2023.

Builds on

87 of 134 references.

References · 87

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2016Layer Normalizationcited by 5,669
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  11. 2017On Calibration of Modern Neural Networkscited by 4,960
  12. 2014Going Deeper with Convolutionscited by 4,241
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  7. 2022Emergent Abilities of Large Language Modelscited by 2,512
  8. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  9. 2013Describing Textures in the Wildcited by 2,125
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  11. 2017The Kinetics Human Action Video Datasetcited by 1,909
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  13. 2021Vision Transformers for Dense Predictioncited by 1,803
  14. 2021ViViT: A Video Vision Transformercited by 1,620
  15. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  16. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  17. 2019Root Mean Square Layer Normalizationcited by 1,480
  18. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  19. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  20. 2019Natural Adversarial Examplescited by 1,408
  21. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  22. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,274
  23. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  24. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  25. 2018Model Cards for Model Reportingcited by 1,080
  26. 2021Scaling Vision Transformerscited by 999
  27. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  28. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  29. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  30. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  31. 2020Quantifying Attention Flow in Transformerscited by 775
  32. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  33. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  34. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  35. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  36. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  37. 2022Efficiently Scaling Transformer Inferencecited by 539
  38. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 164; 2024: 221; 2025: 187; 2026: 122.

694 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  5. 2025Gemma 3 Technical Reportcited by 1,824
  6. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
  7. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  8. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  9. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  10. 2024Genie: Generative Interactive Environmentscited by 747
  11. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  12. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
38 more
  1. 2025Kimi K2: Open Agentic Intelligencecited by 567
  2. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  3. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  4. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  5. 2023Scaling Data-Constrained Language Modelscited by 403
  6. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  7. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  8. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 337
  9. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  10. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  11. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  12. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  13. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 306
  14. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  15. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 305
  16. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  17. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  18. 2023Towards Generalist Biomedical AIcited by 289
  19. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  20. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  21. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 229
  22. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  23. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 211
  24. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  25. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  26. 2023EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representationscited by 204
  27. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  28. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
  29. 2024CBraMod: A Criss-Cross Brain Foundation Model for EEG Decodingcited by 199
  30. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  31. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 185
  32. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  33. 2023RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Modelcited by 181
  34. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  35. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  36. 2024The Road Less Scheduledcited by 163
  37. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 161
  38. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147

50 most cited of 694 All 694 (JSON)