mlsanityData to 7 Oct 2026

A Survey of Vision-Language Pre-Trained Models

arXiv 2202.10936 ↗First posted 18 Feb 2022cs.CV cs.CL cs.LGcited by 177

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

As transformer evolves, pre-trained models have advanced at a breakneck pace in recent years. They have dominated the mainstream techniques in natural language processing (NLP) and computer vision (CV). How to adapt pre-training to the field of Vision-and-Language (V-L) learning and improve downstream task performance becomes a focus of multimodal learning. In this paper, we review the recent progress in Vision-Language Pre-Trained Models (VL-PTMs). As the core content, we first briefly introduce several ways to encode raw images and texts to single-modal embeddings before pre-training. Then, we dive into the mainstream architectures of VL-PTMs in modeling the interaction between text and image representations. We further present widely-used pre-training tasks, and then we introduce some common downstream tasks. We finally conclude this paper and present some promising research directions. Our survey aims to provide researchers with synthesis and pointer to related research.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 61 references that are in the map. First posted on arXiv 18 Feb 2022.

Builds on

48 of 61 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
  2. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  3. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training25 Jun
  4. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
32 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  11. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  12. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  13. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  14. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  15. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  16. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  17. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  18. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  19. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  20. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  21. 2017Gated Multimodal Units for Information Fusioncited by 244
  22. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  23. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  24. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  25. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  26. 2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
  27. 2020X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformerscited by 58
  28. 2019Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methodscited by 55
  29. 2020XGPT: Cross-modal Generative Pre-Training for Image Captioningcited by 51
  30. 2020InterBERT: Vision-and-Language Interaction for Multi-modal Pretrainingcited by 48
  31. 2021ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generationcited by 41
  32. 2021Survey: Transformer based Video-Language Pre-trainingcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 17; 2023: 52; 2024: 51; 2025: 42; 2026: 15.

177 papers.

Sort:

By area:

  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  2. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  3. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  4. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  5. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  6. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  7. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  8. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  9. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  10. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  11. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  12. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
38 more
  1. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 103
  2. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  3. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  4. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  5. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 66
  6. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  7. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  8. 2023A Comprehensive Survey on Segment Anything Model for Vision and Beyondcited by 64
  9. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 59
  10. 2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 57
  11. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  12. 2025GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoningcited by 56
  13. 2024Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Drivingcited by 52
  14. 2023SayCanPay: Heuristic Planning with Large Language Models using Learnable Domain Knowledgecited by 49
  15. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
  16. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  17. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  18. 2023CLIP2Protect: Protecting Facial Privacy using Text-Guided Makeup via Adversarial Latent Searchcited by 42
  19. 2023VideoXum: Cross-modal Visual and Textural Summarization of Videoscited by 41
  20. 2024Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4Vcited by 40
  21. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Modelscited by 39
  22. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  23. 2024Dual Prototype Evolving for Test-Time Generalization of Vision-Language Modelscited by 36
  24. 2023A survey of the Vision Transformers and their CNN-Transformer based Variantscited by 33
  25. 2023EventBind: Learning a Unified Representation to Bind Them All for Event-based Open-world Understandingcited by 33
  26. 2024ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and Morecited by 33
  27. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31
  28. 2022A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainabilitycited by 30
  29. 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Modelscited by 30
  30. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
  31. 2023POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained modelscited by 29
  32. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  33. 2024Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Modelscited by 28
  34. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  35. 2023ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Modelscited by 26
  36. 2024GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discoverycited by 26
  37. 2023Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrievalcited by 24
  38. 2023Large Pre-trained time series models for cross-domain Time series analysis taskscited by 22

50 most cited of 177 All 177 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022VLP: A Survey on Vision-Language Pre-training
  2. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
  3. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  4. 2023Vision Language Transformers: A Survey
  5. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  6. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  7. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey
  8. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels