mlsanityData to 7 Oct 2026

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

arXiv 2010.11929 ↗First posted 22 Oct 2020cs.CV cs.AI cs.LGcited by 29,915

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.

Key facts. Cited by 29,915 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 1 of its 58 references that are in the map. First posted on arXiv 22 Oct 2020.

Builds on

1 of 58 references.

References · 1

  1. 2020Language Models are Few-Shot Learnerscited by 32,199

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1453; 2022: 3504; 2023: 5734; 2024: 7522; 2025: 7072; 2026: 4630.

29,915 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2023Segment Anythingcited by 8,158
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  6. 2022A ConvNet for the 2020scited by 3,545
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  13. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  14. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  15. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  16. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  17. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  18. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  19. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  20. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  21. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  22. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  23. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  24. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,869
  25. 2021Vision Transformers for Dense Predictioncited by 1,789
  26. 2025Gemma 3 Technical Reportcited by 1,686
  27. 2022Visual Prompt Tuningcited by 1,674
  28. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  29. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,618
  30. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  31. 2021ViViT: A Video Vision Transformercited by 1,610
  32. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  33. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  34. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  35. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  36. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  37. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  38. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449

50 most cited of 29,915 All 29,915 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SKDU at De-Factify 4.0: Vision Transformer with Data Augmentation for AI-Generated Image Detection
  2. 2022SaiT: Sparse Vision Transformers through Adaptive Token Pruning
  3. 2023COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Models
  4. 2022Exploring Transformer Backbones for Image Diffusion Models
  5. 2023Multi-Scale And Token Mergence: Make Your ViT More Efficient
  6. 2025On the Limitations of Vision-Language Models in Understanding Image Transforms
  7. 2026ResLRP: The Role of Residual Cancellation in Attribution Instability in Vision Transformers
  8. 2022Exploring Vision Transformers as Diffusion Learners