mlsanityData to 7 Oct 2026

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

arXiv 2210.09263 ↗First posted 17 Oct 2022cs.CV cs.CLcited by 117

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

This paper surveys vision-language pre-training (VLP) methods for multimodal intelligence that have been developed in the last few years. We group these approaches into three categories: (i) VLP for image-text tasks, such as image captioning, image-text retrieval, visual question answering, and visual grounding; (ii) VLP for core computer vision tasks, such as (open-set) image classification, object detection, and segmentation; and (iii) VLP for video-text tasks, such as video captioning, video-text retrieval, and video question answering. For each category, we present a comprehensive review of state-of-the-art methods, and discuss the progress that has been made and challenges still being faced, using specific systems and models as case studies. In addition, for each category, we discuss advanced topics being actively explored in the research community, such as big foundation models, unified modeling, in-context few-shot learning, knowledge, robustness, and computer vision in the wild, to name a few.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 300 of its 533 references that are in the map. First posted on arXiv 17 Oct 2022.

Builds on

300 of 533 references.

Closest ancestors · 15

Same direction, newest first

  1. 2022Write and Paint: Generative Vision-Language Models are Unified Modal Learners15 Jun
  2. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone15 Jun
  3. 2022VL-BEiT: Generative Vision-Language Pretraining2 Jun
  4. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models30 May
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  6. 2022VLP: A Survey on Vision-Language Pre-training18 Feb
  7. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark14 Feb
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  9. 2021Scaling Up Vision-Language Pre-training for Image Captioning24 Nov
  10. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov
  11. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training9 Nov
  12. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
3 more
  1. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  2. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training25 Jun
  3. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 285

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  12. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2018YOLOv3: An Incremental Improvementcited by 2,652
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  13. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  14. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  15. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  16. 2017The Kinetics Human Action Video Datasetcited by 1,894
  17. 2018SlowFast Networks for Video Recognitioncited by 1,862
  18. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  19. 2021Vision Transformers for Dense Predictioncited by 1,789
  20. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  21. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  22. 2019Towards VQA Models That Can Readcited by 1,696
  23. 2021ViViT: A Video Vision Transformercited by 1,610
  24. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  25. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  26. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  27. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  28. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  29. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  30. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  31. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  32. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  33. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  34. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  35. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  36. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  37. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  38. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114

50 most cited of 285 All 285 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 49; 2024: 34; 2025: 22; 2026: 8.

117 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  6. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  7. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  8. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  9. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  10. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
  11. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  12. 2023A Survey on Video Diffusion Modelscited by 203
38 more
  1. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  2. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  3. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  4. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  5. 2023Large Language Models for Robotics: A Surveycited by 132
  6. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  7. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  8. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  9. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 85
  10. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  11. 2024Compositional Entailment Learning for Hyperbolic Vision-Language Modelscited by 72
  12. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuningcited by 69
  13. 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
  14. 2023A Billion-scale Foundation Model for Remote Sensing Imagescited by 62
  15. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  16. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  17. 2024Toward Visual Grounding: A Surveycited by 52
  18. 2022Multitask Vision-Language Prompt Tuningcited by 51
  19. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 51
  20. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  21. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 41
  22. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  23. 2023Language Models as Black-Box Optimizers for Vision-Language Modelscited by 36
  24. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 35
  25. 2024On the Challenges and Opportunities in Generative AIcited by 31
  26. 2022MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybridcited by 29
  27. 2023CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservationcited by 29
  28. 2024CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarioscited by 29
  29. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  30. 2023A Review of Deep Learning for Video Captioningcited by 28
  31. 2023Bootstrapping Vision-Language Learning with Decoupled Language Pre-trainingcited by 28
  32. 2023What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuningcited by 28
  33. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 28
  34. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  35. 2024Foundation Models in Radiology: What, How, When, Why and Why Notcited by 26
  36. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  37. 2023Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Modelscited by 22
  38. 2023Extending Multi-modal Contrastive Representationscited by 21

50 most cited of 117 All 117 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022A Survey of Vision-Language Pre-Trained Models
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  3. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey
  4. 2023A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translation
  5. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations
  6. 2023Vision-Language Models for Vision Tasks: A Survey
  7. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  8. 2024Probing Visual Language Priors in VLMs