mlsanityData to 7 Oct 2026

Vision Language Transformers: A Survey

arXiv 2307.03254 ↗First posted 6 Jul 2023cs.CV cs.AI cs.CLcited by 5

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision language tasks, such as answering questions about or generating captions that describe an image, are difficult tasks for computers to perform. A relatively recent body of research has adapted the pretrained transformer architecture introduced in citetvaswani2017attention to vision language modeling. Transformer models have greatly improved performance and versatility over previous vision language models. They do so by pretraining models on a large generic datasets and transferring their learning to new tasks with minor changes in architecture and parameter values. This type of transfer learning has become the standard modeling practice in both natural language processing and computer vision. Vision language transformers offer the promise of producing similar advancements in tasks which require both vision and language. In this paper, we provide a broad synthesis of the currently available research on vision language transformer models and offer some analysis of their strengths, limitations and some open questions that remain.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 83 of its 95 references that are in the map. First posted on arXiv 6 Jul 2023.

Builds on

83 of 95 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022Position-guided Text Prompt for Vision-Language Pre-training19 Dec
  2. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks22 Nov
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  4. 2021Scaling Up Vision-Language Pre-training for Image Captioning24 Nov
  5. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov
  6. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
  7. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  8. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning3 Jun
  9. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  11. 2020End-to-End Object Detection with Transformerscited by 7,384
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2015Fast R-CNNcited by 2,930
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  13. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  14. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  15. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  16. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  17. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  18. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  19. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  20. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  21. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  22. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  23. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  24. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  25. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  26. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  27. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  28. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  29. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  30. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  31. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  32. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  33. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  34. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  35. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  36. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  37. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  38. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 2.

5 papers.

  1. 2025Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answeringcited by 7
  2. 2024Transformers Utilization in Chart Understanding: A Review of Recent Advances & Future Trendscited by 3
  3. 2025Prior Lessons of Incremental Dialogue and Robot Action Management for the Age of Language Modelscited by 2
  4. 2023On the Computational Modeling of Meaning: Embodied Cognition Intertwined with Emotioncited by 1
  5. 2024Performance Evaluation of Deep Learning and Transformer Models Using Multimodal Data for Breast Cancer Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey
  2. 2022A Survey of Vision-Language Pre-Trained Models
  3. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  4. 2025On the Limitations of Vision-Language Models in Understanding Image Transforms
  5. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  6. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
  7. 2024An Introduction to Vision-Language Modeling
  8. 2025Improved Alignment of Modalities in Large Vision Language Models