mlsanityData to 7 Oct 2026

Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions

arXiv 2404.07214 ↗First posted 20 Feb 2024cs.CV cs.AI cs.CLcited by 66

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The advent of Large Language Models (LLMs) has significantly reshaped the trajectory of the AI revolution. Nevertheless, these LLMs exhibit a notable limitation, as they are primarily adept at processing textual information. To address this constraint, researchers have endeavored to integrate visual capabilities with LLMs, resulting in the emergence of Vision-Language Models (VLMs). These advanced models are instrumental in tackling more intricate tasks such as image captioning and visual question answering. In our comprehensive survey paper, we delve into the key advancements within the realm of VLMs. Our classification organizes VLMs into three distinct categories: models dedicated to vision-language understanding, models that process multimodal inputs to generate unimodal (textual) outputs and models that both accept and produce multimodal inputs and outputs.This classification is based on their respective capabilities and functionalities in processing and generating various modalities of data.We meticulously dissect each model, offering an extensive analysis of its foundational architecture, training data sources, as well as its strengths and limitations wherever possible, providing readers with a comprehensive understanding of its essential components. We also analyzed the performance of VLMs in various benchmark datasets. By doing so, we aim to offer a nuanced understanding of the diverse landscape of VLMs. Additionally, we underscore potential avenues for future research in this dynamic domain, anticipating further breakthroughs and advancements.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 94 of its 124 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

94 of 124 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objects8 Dec
  2. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback16 Nov
  3. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  5. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions19 Aug
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  9. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
  10. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 84

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
38 more
  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  2. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  3. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  4. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  6. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  7. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  8. 2021Grounded Language-Image Pre-trainingcited by 1,147
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2023A Survey on Multimodal Large Language Modelscited by 898
  12. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  13. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  14. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  15. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  16. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  17. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  18. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  19. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  20. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  21. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  22. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  23. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  24. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  25. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  26. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  27. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  28. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  29. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  30. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  31. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  32. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  33. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  34. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  35. 2023Demystifying CLIP Datacited by 265
  36. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  37. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  38. 2023Any-to-Any Generation via Composable Diffusioncited by 203

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 46; 2026: 14.

66 papers.

Sort:

By area:

  1. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  2. 2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 57
  3. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 42
  4. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  5. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31
  6. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Modelscited by 31
  7. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  8. 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustnesscited by 28
  9. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  10. 2025A Survey on Efficient Vision-Language Modelscited by 19
  11. 2025Towards Visual Text Grounding of Multimodal Large Language Modelcited by 18
  12. 2025AutoNeural: Co-Designing Vision-Language Models for NPU Inferencecited by 17
38 more
  1. 2024Do Language Models Understand Time?cited by 14
  2. 2025Benchmarking Foundation Models for Zero-Shot Biometric Taskscited by 13
  3. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 13
  4. 2025Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Modelscited by 12
  5. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  6. 2024GameVLM: A Decision-making Framework for Robotic Task Planning Based on Visual Language Models and Zero-sum Gamescited by 7
  7. 2025Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answeringcited by 7
  8. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  9. 2024H-POPE: Hierarchical Polling-based Probing Evaluation of Hallucinations in Large Vision-Language Modelscited by 6
  10. 2025From Shots to Stories: LLM-Assisted Video Editing with Unified Language Representationscited by 6
  11. 2025Empowering Small VLMs to Think with Dynamic Memorization and Explorationcited by 5
  12. 2025Vision-Language Models display a strong gender biascited by 5
  13. 2026APEX: A Decoupled Memory-based Explorer for Asynchronous Aerial Object Goal Navigationcited by 5
  14. 2025Vision language models are unreliable at trivial spatial cognitioncited by 4
  15. 2025MemeSense: An Adaptive In-Context Framework for Social Commonsense Driven Meme Moderationcited by 3
  16. 2025Window Token Concatenation for Efficient Visual Large Language Modelscited by 3
  17. 2025VLLFL: A Vision-Language Model Based Lightweight Federated Learning Framework for Smart Agriculturecited by 3
  18. 2025TextBO: Bayesian Optimization in Language Space for Eval-Efficient Self-Improving AIcited by 3
  19. 2026CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcarecited by 3
  20. 2026White-Balance First, Adjust Later: Cross-Camera Color Constancy via Vision-Language Evaluationcited by 3
  21. 2024Do large language vision models understand 3D shapes?cited by 2
  22. 2025Playing Devil's Advocate: Unmasking Toxicity and Vulnerabilities in Large Vision-Language Modelscited by 2
  23. 2025NanoVLMs: How small can we go and still make coherent Vision Language Models?cited by 2
  24. 2025Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership Inference Privacy Leakage?cited by 2
  25. 2025Exploring Fusion Strategies for Multimodal Vision-Language Systemscited by 2
  26. 2024Imperfect Vision Encoders: Efficient and Robust Tuning for Vision-Language Modelscited by 1
  27. 2025Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UIcited by 1
  28. 2025Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Trainingcited by 1
  29. 2025Underwater Diffusion Attention Network with Contrastive Language-Image Joint Learning for Underwater Image Enhancementcited by 1
  30. 2025FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Modelscited by 1
  31. 2025LumiGen: An LVLM-Enhanced Iterative Framework for Fine-Grained Text-to-Image Generationcited by 1
  32. 2025Text-conditioned State Space Model For Domain-generalized Change Detection Visual Question Answeringcited by 1
  33. 2025CMRAG: Co-modality-based visual document retrieval and question answeringcited by 1
  34. 2025Artificial Phantasia: Emergent Mental Imagery in Large Language Modelscited by 1
  35. 2025Leveraging Text-Driven Semantic Variation for Robust OOD Segmentationcited by 1
  36. 2025MMRPT: MultiModal Reinforcement Pre-Training via Masked Vision-Dependent Reasoningcited by 1
  37. 2025Reasoning Palette: Modulating Reasoning via Latent Contextualization for Controllable Exploration for (V)LMscited by 1
  38. 2026Investigating the Viability of Employing Multi-modal Large Language Models in the Context of Audio Deepfake Detectioncited by 1

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  2. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  3. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  4. 2024An Introduction to Vision-Language Modeling
  5. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey
  6. 2024Building and better understanding vision-language models: insights and future directions
  7. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
  8. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends