mlsanityData to 7 Oct 2026

Vision-Language Models for Vision Tasks: A Survey

arXiv 2304.00685 ↗First posted 3 Apr 2023cs.CVcited by 834

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Most visual recognition studies rely heavily on crowd-labelled data in deep neural networks (DNNs) training, and they usually train a DNN for each single visual recognition task, leading to a laborious and time-consuming visual recognition paradigm. To address the two challenges, Vision-Language Models (VLMs) have been intensively investigated recently, which learns rich vision-language correlation from web-scale image-text pairs that are almost infinitely available on the Internet and enables zero-shot predictions on various visual recognition tasks with a single VLM. This paper provides a systematic review of visual language models for various visual recognition tasks, including: (1) the background that introduces the development of visual recognition paradigms; (2) the foundations of VLM that summarize the widely-adopted network architectures, pre-training objectives, and downstream tasks; (3) the widely-adopted datasets in VLM pre-training and evaluations; (4) the review and categorization of existing VLM pre-training methods, VLM transfer learning methods, and VLM knowledge distillation methods; (5) the benchmarking, analysis and discussion of the reviewed methods; (6) several research challenges and potential research directions that could be pursued in the future VLM studies for visual recognition. A project associated with this survey has been created at https://github.com/jingyi0000/VLM_survey.

Key facts. Cited by 834 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 168 of its 233 references that are in the map. First posted on arXiv 3 Apr 2023.

Builds on

168 of 233 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone15 Jun
  2. 2022A Survey of Vision-Language Pre-Trained Models18 Feb
  3. 2022VLP: A Survey on Vision-Language Pre-training18 Feb
  4. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training9 Nov
  5. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov

Other references · 163

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  11. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  12. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
38 more
  1. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  13. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  14. 2015Fast R-CNNcited by 2,930
  15. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  16. 2020Supervised Contrastive Learningcited by 2,803
  17. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  18. 2015VQA: Visual Question Answeringcited by 2,489
  19. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  20. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  21. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  22. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  23. 2013Describing Textures in the Wildcited by 2,110
  24. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  25. 2022Visual Prompt Tuningcited by 1,674
  26. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  27. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  28. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  29. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  30. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  31. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  32. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  33. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  34. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  35. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  36. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  37. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  38. 2021Grounded Language-Image Pre-trainingcited by 1,147

50 most cited of 163 All 163 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 37; 2024: 192; 2025: 373; 2026: 232.

834 papers.

Sort:

By area:

  1. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  2. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  3. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  4. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  5. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  6. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  7. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  8. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  9. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  10. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  11. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  12. 2023Controlling Vision-Language Models for Multi-Task Image Restorationcited by 108
38 more
  1. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  2. 2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 107
  3. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  4. 2025TerraMind: Large-Scale Generative Multimodality for Earth Observationcited by 101
  5. 2025Visual Planning: Let's Think Only with Imagescited by 85
  6. 2023The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shotcited by 83
  7. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
  8. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  9. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  10. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  11. 2024TableGPT2: A Large Multimodal Model with Tabular Data Integrationcited by 69
  12. 2025KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Modelscited by 69
  13. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  14. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 65
  15. 2024Prompt Compression for Large Language Models: A Surveycited by 65
  16. 2025HVI: A New Color Space for Low-light Image Enhancementcited by 62
  17. 2024GraphCLIP: Enhancing Transferability in Graph Foundation Models for Text-Attributed Graphscited by 58
  18. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Modelcited by 57
  19. 2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 57
  20. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 57
  21. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  22. 2024Jailbreak Large Vision-Language Models Through Multi-Modal Linkagecited by 53
  23. 2024KeyVideoLLM: Towards Large-scale Video Keyframe Selectioncited by 52
  24. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 52
  25. 2024Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMscited by 48
  26. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
  27. 2024Position: What Can Large Language Models Tell Us about Time Series Analysiscited by 47
  28. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  29. 2023CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcarecited by 43
  30. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  31. 2023The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classificationcited by 41
  32. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  33. 2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 40
  34. 2024A Sober Look at the Robustness of CLIPs to Spurious Featurescited by 38
  35. 2024Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Makingcited by 38
  36. 2024VistaDream: Sampling multiview consistent images for single-view scene reconstructioncited by 38
  37. 2025RemoteSAM: Towards Segment Anything for Earth Observationcited by 38
  38. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 37

50 most cited of 834 All 834 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024An Introduction to Vision-Language Modeling
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  3. 2025Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
  5. 2024Rethinking VLMs and LLMs for Image Classification
  6. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluation
  7. 2021VinVL: Revisiting Visual Representations in Vision-Language Models
  8. 2024Response Wide Shut: Surprising Observations in Basic Vision Language Model Capabilities