mlsanityData to 14 Oct 2026

An Introduction to Vision-Language Modeling

arXiv 2405.17247 ↗First posted 27 May 2024cs.LGcited by 178

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Following the recent popularity of Large Language Models (LLMs), several attempts have been made to extend them to the visual domain. From having a visual assistant that could guide us through unfamiliar environments to generative models that produce images using only a high-level text description, the vision-language model (VLM) applications will significantly impact our relationship with technology. However, there are many challenges that need to be addressed to improve the reliability of those models. While language is discrete, vision evolves in a much higher dimensional space in which concepts cannot always be easily discretized. To better understand the mechanics behind mapping vision to language, we present this introduction to VLMs which we hope will help anyone who would like to enter the field. First, we introduce what VLMs are, how they work, and how to train them. Then, we present and discuss approaches to evaluate VLMs. Although this work primarily focuses on mapping images to language, we also discuss extending VLMs to videos.

Key facts. Cited by 178 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 82 of its 298 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

82 of 298 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024D'ej`a Vu Memorization in Vision-Language Models3 Feb
  2. 2024The Neglected Tails in Vision-Language Models23 Jan
  3. 2023GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language models15 Nov
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  5. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  6. 2023Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering16 Jun
  7. 2023Revisiting the Role of Language Priors in Vision-Language Models2 Jun
  8. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations1 Jul

Other references · 74

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  11. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
38 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  2. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  3. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  5. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  6. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  8. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  9. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  11. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  12. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  13. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  14. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  15. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  16. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  17. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  18. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  19. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  20. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 279
  21. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  22. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  23. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 265
  24. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 236
  25. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  26. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 170
  27. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  28. 2022Holistic Evaluation of Language Modelscited by 131
  29. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  30. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokenscited by 118
  31. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 108
  32. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 105
  33. 2024How to Train Data-Efficient LLMscited by 93
  34. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  35. 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 76
  36. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  37. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  38. 2024Red Teaming Visual Language Modelscited by 65

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 36; 2025: 94; 2026: 48.

178 papers.

Sort:

By area:

  1. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Modelscited by 118
  2. 2025TerraMind: Large-Scale Generative Multimodality for Earth Observationcited by 104
  3. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  4. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 74
  5. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
  6. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 57
  7. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Modelcited by 57
  8. 2025IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environmentscited by 52
  9. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  10. 2025TLA: Tactile-Language-Action Model for Contact-Rich Manipulationcited by 47
  11. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 40
  12. 2024INQUIRE: A Natural World Text-to-Image Retrieval Benchmarkcited by 33
38 more
  1. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scalingcited by 31
  2. 2024What is the Visual Cognition Gap between Humans and Multimodal LLMs?cited by 30
  3. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  4. 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMscited by 25
  5. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 22
  6. 2025Benchmarking Visual Language Models on Standardized Visualization Literacy Testscited by 20
  7. 2024Fine-Tuning Vision-Language Model for Automated Engineering Drawing Information Extractioncited by 18
  8. 2024Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documentscited by 17
  9. 2025Tokenizing Single-Channel EEG with Time-Frequency Motif Learningcited by 17
  10. 2025OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understandingcited by 17
  11. 2024Enabling Novel Mission Operations and Interactions with ROSA: The Robot Operating System Agentcited by 16
  12. 2024OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalitiescited by 16
  13. 2024Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?cited by 16
  14. 2025Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Drivingcited by 16
  15. 2024Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Modelscited by 15
  16. 2024Promoting cross-modal representations to improve multimodal foundation models for physiological signalscited by 13
  17. 2024Post-hoc Probabilistic Vision-Language Modelscited by 13
  18. 2024WalkVLM:Aid Visually Impaired People Walking by Vision Language Modelcited by 13
  19. 2025Using Vision Language Models for Safety Hazard Identification in Constructioncited by 13
  20. 2025Object-centric Binding in Contrastive Language-Image Pretrainingcited by 12
  21. 2025Vision Language Models in Medicinecited by 12
  22. 2024LLaVA Needs More Knowledge: Retrieval Augmented Natural Language Generation with Knowledge Graph for Explaining Thoracic Pathologiescited by 11
  23. 2024Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice Question Answering by Video Language Modelscited by 11
  24. 2025Texture or Semantics? Vision-Language Models Get Lost in Font Recognitioncited by 11
  25. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 11
  26. 2024Multimodal Fact-Checking with Vision Language Models: A Probing Classifier based Solution with Embedding Strategiescited by 10
  27. 2025Levels of Analysis for Large Language Modelscited by 10
  28. 2026V-Zero: Self-Improving Multimodal Reasoning with Zero Annotationcited by 10
  29. 2024Have Large Vision-Language Models Mastered Art History?cited by 9
  30. 2025One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Imagecited by 8
  31. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 8
  32. 2025Fundamentals of Building Autonomous LLM Agentscited by 8
  33. 2024Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical Reportcited by 7
  34. 2025Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognitioncited by 7
  35. 2025Are Vision-Language Models Ready for Dietary Assessment? Exploring the Next Frontier in AI-Powered Food Image Recognitioncited by 7
  36. 2025Domain Adaptation of VLM for Soccer Video Understandingcited by 7
  37. 2025DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understandingcited by 7
  38. 2026What, Whether and How? Unveiling Process Reward Models for Thinking with Images Reasoningcited by 7

50 most cited of 178 All 178 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Visual Large Language Models for Generalized and Specialized Applications
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  3. 2024Building and better understanding vision-language models: insights and future directions
  4. 2025Hidden in plain sight: VLMs overlook their visual representations
  5. 2025Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities
  6. 2024What matters when building vision-language models?
  7. 2025A Survey on Efficient Vision-Language Models
  8. 2023Vision-Language Models for Vision Tasks: A Survey