mlsanityData to 14 Oct 2026

Building and better understanding vision-language models: insights and future directions

arXiv 2408.12637 ↗First posted 22 Aug 2024cs.CV cs.AIcited by 173

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The field of vision-language models (VLMs), which take images and texts as inputs and output texts, is rapidly evolving and has yet to reach consensus on several key aspects of the development pipeline, including data, architecture, and training methods. This paper can be seen as a tutorial for building a VLM. We begin by providing a comprehensive overview of the current state-of-the-art approaches, highlighting the strengths and weaknesses of each, addressing the major challenges in the field, and suggesting promising research directions for underexplored areas. We then walk through the practical steps to build Idefics3-8B, a powerful VLM that significantly outperforms its predecessor Idefics2-8B, while being trained efficiently, exclusively on open datasets, and using a straightforward pipeline. These steps include the creation of Docmatix, a dataset for improving document understanding capabilities, which is 240 times larger than previously available datasets. We release the model along with the datasets created for its training.

Key facts. Cited by 173 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 182 of its 213 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

182 of 213 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  2. 2024From Pixels to Prose: A Large Dataset of Dense Image Captions14 Jun
  3. 2024What matters when building vision-language models?3 May
  4. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  5. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models27 Mar
  6. 2024Synth2: Boosting Visual-Language Models with Synthetic Captions and Image Embeddings12 Mar
  7. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Model6 Feb
  8. 2023PaLI-3 Vision Language Models: Smaller, Faster, Stronger13 Oct
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  11. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
1 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 169

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  11. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  12. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
38 more
  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2023Mistral 7Bcited by 3,164
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  13. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  14. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  15. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  16. 2015VQA: Visual Question Answeringcited by 2,506
  17. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  18. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  19. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  20. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  21. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  22. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  23. 2019Towards VQA Models That Can Readcited by 1,718
  24. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  25. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  26. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  27. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  28. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  29. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  30. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  31. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  32. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  33. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  34. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  35. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  36. 2023LIMA: Less Is More for Alignmentcited by 1,100
  37. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  38. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050

50 most cited of 169 All 169 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 28; 2025: 89; 2026: 56.

173 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  3. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  4. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  5. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
  6. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  7. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 175
  8. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 173
  9. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 170
  10. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 116
  11. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 89
  12. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 81
38 more
  1. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 77
  2. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  3. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  4. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 70
  5. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 59
  6. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 53
  7. 2025FineVision: Open Data Is All You Needcited by 53
  8. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  9. 2025LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?cited by 49
  10. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 41
  11. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  12. 2024Core Knowledge Deficits in Multi-Modal Language Modelscited by 38
  13. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Trainingcited by 38
  14. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  15. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 32
  16. 2025Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluationcited by 30
  17. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  18. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
  19. 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 27
  20. 2025RadVLM: A Multitask Conversational Vision-Language Model for Radiologycited by 25
  21. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 25
  22. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understandingcited by 23
  23. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 22
  24. 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Modelscited by 21
  25. 2025ModernVBERT: Towards Smaller Visual Document Retrieverscited by 21
  26. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
  27. 2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
  28. 2025ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmarkcited by 19
  29. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  30. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  31. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Taskscited by 18
  32. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectivescited by 18
  33. 2024POINTS: Improving Your Vision-language Model with Affordable Strategiescited by 17
  34. 2024LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Contentcited by 17
  35. 2025NVIDIA Nemotron Nano V2 VLcited by 17
  36. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  37. 2025Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Searchcited by 16
  38. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 16

50 most cited of 173 All 173 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  2. 2024An Introduction to Vision-Language Modeling
  3. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models
  4. 2026SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding
  5. 2025Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality
  6. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding
  7. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  8. 2026ReViCo: Unveiling the Limitations of VLMs in Visual Text Understanding via Error Correction