mlsanityData to 7 Oct 2026

VLP: A Survey on Vision-Language Pre-training

arXiv 2202.09061 ↗First posted 18 Feb 2022cs.CV cs.CLcited by 159

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

In the past few years, the emergence of pre-training models has brought uni-modal fields such as computer vision (CV) and natural language processing (NLP) to a new era. Substantial works have shown they are beneficial for downstream uni-modal tasks and avoid training a new model from scratch. So can such pre-trained models be applied to multi-modal tasks? Researchers have explored this problem and made significant progress. This paper surveys recent advances and new frontiers in vision-language pre-training (VLP), including image-text and video-text pre-training. To give readers a better overall grasp of VLP, we first review its recent advances from five aspects: feature extraction, model architecture, pre-training objectives, pre-training datasets, and downstream tasks. Then, we summarize the specific VLP models in detail. Finally, we discuss the new frontiers in VLP. To the best of our knowledge, this is the first survey focused on VLP. We hope that this survey can shed light on future research in the VLP field.

Key facts. Cited by 159 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 123 of its 168 references that are in the map. First posted on arXiv 18 Feb 2022.

Builds on

123 of 168 references.

Closest ancestors · 6

Same direction, newest first

  1. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov
  2. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
  3. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  4. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training25 Jun
  5. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning3 Jun
  6. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 117

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
38 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2018SlowFast Networks for Video Recognitioncited by 1,862
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  12. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  13. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  14. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  15. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  16. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  17. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  18. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  19. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  20. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  21. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  22. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  23. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  24. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  25. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  26. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  27. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  28. 2020A Survey on Visual Transformercited by 653
  29. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  30. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  31. 2018Object Detection with Deep Learning: A Reviewcited by 524
  32. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  33. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  34. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  35. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  36. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  37. 2018nocaps: novel object captioning at scalecited by 444
  38. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434

50 most cited of 117 All 117 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 41; 2024: 56; 2025: 32; 2026: 18.

159 papers.

Sort:

By area:

  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  3. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  4. 2022Multimodal Learning with Transformers: A Surveycited by 399
  5. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  6. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  7. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  9. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  10. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  11. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  12. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
38 more
  1. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  2. 2023VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detectioncited by 121
  3. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  4. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  5. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  6. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  7. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  8. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  9. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  10. 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 72
  11. 2023A Comprehensive Survey on Segment Anything Model for Vision and Beyondcited by 64
  12. 2023A Billion-scale Foundation Model for Remote Sensing Imagescited by 62
  13. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 54
  14. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Promptscited by 51
  15. 2023UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generationcited by 47
  16. 2024GeoEval: Benchmark for Evaluating LLMs and Multi-Modal Models on Geometry Problem-Solvingcited by 44
  17. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentationcited by 39
  18. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Modelscited by 39
  19. 2023Federated Incremental Semantic Segmentationcited by 36
  20. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimizationcited by 36
  21. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 35
  22. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captionscited by 33
  23. 2023EventBind: Learning a Unified Representation to Bind Them All for Event-based Open-world Understandingcited by 33
  24. 2024ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and Morecited by 33
  25. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuningcited by 33
  26. 2024TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Datacited by 32
  27. 2024Review of multimodal machine learning approaches in healthcarecited by 30
  28. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  29. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  30. 2024Libra: Leveraging Temporal Images for Biomedical Radiology Analysiscited by 29
  31. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  32. 2024C-RAG: Certified Generation Risks for Retrieval-Augmented Language Modelscited by 27
  33. 2024GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discoverycited by 26
  34. 2022Bidirectional Generation of Structure and Properties Through a Single Molecular Foundation Modelcited by 25
  35. 2023Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrievalcited by 24
  36. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  37. 2024MouSi: Poly-Visual-Expert Vision-Language Modelscited by 24
  38. 2025SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMscited by 23

50 most cited of 159 All 159 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations
  2. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  3. 2022A Survey of Vision-Language Pre-Trained Models
  4. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models
  5. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey
  6. 2021Grid-VLP: Revisiting Grid Features for Vision-Language Pre-training
  7. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
  8. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions