mlsanityData to 7 Oct 2026

Vision-and-Language Pretrained Models: A Survey

arXiv 2204.07356 ↗First posted 15 Apr 2022cs.CV cs.CLcited by 50

Not on the map. Why?

Abstract (arXiv)

Pretrained models have produced great success in both Computer Vision (CV) and Natural Language Processing (NLP). This progress leads to learning joint representations of vision and language pretraining by feeding visual and linguistic contents into a multi-layer transformer, Visual-Language Pretrained Models (VLPMs). In this paper, we present an overview of the major advances achieved in VLPMs for producing joint representations of vision and language. As the preliminaries, we briefly describe the general task definition and genetic architecture of VLPMs. We first discuss the language and vision data encoding methods and then present the mainstream VLPM structure as the core content. We further summarise several essential pretraining and fine-tuning strategies. Finally, we highlight three future directions for both CV and NLP researchers to provide insightful guidance.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 33 of its 41 references that are in the map. First posted on arXiv 15 Apr 2022.

Builds on

33 of 41 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  11. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  12. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
21 more
  1. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  2. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  3. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  4. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  5. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  6. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  7. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  8. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  9. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  10. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  11. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  12. 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95
  13. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  14. 2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
  15. 2019Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baselinecited by 60
  16. 2019Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methodscited by 55
  17. 2020FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrievalcited by 55
  18. 2020VD-BERT: A Unified Vision and Dialog Transformer with BERTcited by 54
  19. 2020XGPT: Cross-modal Generative Pre-Training for Image Captioningcited by 51
  20. 2021LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrievalcited by 44
  21. 2021Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Networkcited by 17

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 20; 2024: 15; 2025: 9.

50 papers.

Sort:

By area:

  1. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  2. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  3. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  4. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  5. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  6. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  7. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  8. 2022Universal Prompt Tuning for Graph Neural Networkscited by 100
  9. 2023A Comprehensive Survey on Segment Anything Model for Vision and Beyondcited by 64
  10. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Biascited by 57
  11. 2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 57
  12. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
38 more
  1. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  2. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  3. 2023Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attributioncited by 34
  4. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  5. 2023Natural Language Interfaces for Tabular Data Querying and Visualization: A Surveycited by 27
  6. 2025MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answeringcited by 18
  7. 2022Doc-GCN: Heterogeneous Graph Convolutional Networks for Document Layout Analysiscited by 17
  8. 2025Vision Function Layer in Multimodal LLMscited by 17
  9. 2023A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Cluescited by 11
  10. 2023Adventures of Trustworthy Vision-Language Models: A Surveycited by 10
  11. 2024RankCLIP: Ranking-Consistent Language-Image Pretrainingcited by 9
  12. 2023PSP: Pre-Training and Structure Prompt Tuning for Graph Neural Networkscited by 8
  13. 2024CIC: A Framework for Culturally-Aware Image Captioningcited by 8
  14. 2024PDF-MVQA: A Dataset for Multimodal Information Retrieval in PDF-based Visual Question Answeringcited by 7
  15. 2024Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyondcited by 7
  16. 2022VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Modelscited by 6
  17. 2024Do More Details Always Introduce More Hallucinations in LVLM-based Image Captioning?cited by 6
  18. 2024A Large-Scale Study of Model Integration in ML-Enabled Software Systemscited by 6
  19. 2024CATSplat: Context-Aware Transformer with Spatial Guidance for Generalizable 3D Gaussian Splatting from A Single-View Imagecited by 6
  20. 2023Large Language Models Meet Computer Vision: A Brief Surveycited by 5
  21. 2023DSAP: Analyzing Bias Through Demographic Comparison of Datasetscited by 5
  22. 2024Segment Anything for Videos: A Systematic Surveycited by 5
  23. 2022Understanding Attention for Vision-and-Language Taskscited by 4
  24. 2023ROSGPT_Vision: Commanding Robots Using Only Language Models' Promptscited by 4
  25. 2023A Survey on Image-text Multimodal Modelscited by 4
  26. 2023A Diffusion Model for Event Skeleton Generationcited by 3
  27. 2024Younger: The First Dataset for Artificial Intelligence-Generated Neural Network Architecturecited by 3
  28. 2022PiggyBack: Pretrained Visual Question Answering Environment for Backing up Non-deep Learning Professionalscited by 2
  29. 2022SceneGATE: Scene-Graph based co-Attention networks for TExt visual question answeringcited by 2
  30. 2025PASTA: Part-Aware Sketch-to-3D Shape Generation with Text-Aligned Priorcited by 2
  31. 2023Multimodal Understanding Through Correlation Maximization and Minimizationcited by 1
  32. 2023Prototypical Contrastive Transfer Learning for Multimodal Language Understandingcited by 1
  33. 2024Feature Distribution Shift Mitigation with Contrastive Pretraining for Intrusion Detectioncited by 1
  34. 2024VISTA: A Visual and Textual Attention Dataset for Interpreting Multimodal Modelscited by 1
  35. 2025Measuring and Mitigating Hallucinations in Vision-Language Dataset Generation for Remote Sensingcited by 1
  36. 2023ProgSG: Cross-Modality Representation Learning for Programs in Electronic Design Automationcited by 0
  37. 2024InFiConD: Interactive No-code Fine-tuning with Concept-based Knowledge Distillationcited by 0
  38. 2025A Framework for Generating Artificial Datasets to Validate Absolute and Relative Position Conceptscited by 0