mlsanityData to 7 Oct 2026

What matters when building vision-language models?

arXiv 2405.02246 ↗First posted 3 May 2024cs.CV cs.AIcited by 321

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The growing interest in vision-language models (VLMs) has been driven by improvements in large language models and vision transformers. Despite the abundance of literature on this subject, we observe that critical decisions regarding the design of VLMs are often not justified. We argue that these unsupported decisions impede progress in the field by making it difficult to identify which choices improve model performance. To address this issue, we conduct extensive experiments around pre-trained models, architecture choice, data, and training methods. Our consolidation of findings includes the development of Idefics2, an efficient foundational VLM of 8 billion parameters. Idefics2 achieves state-of-the-art performance within its size category across various multimodal benchmarks, and is often on par with models four times its size. We release the model (base, instructed, and chat) along with the datasets created for its training.

Key facts. Cited by 321 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 135 of its 156 references that are in the map. First posted on arXiv 3 May 2024.

Builds on

135 of 156 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models27 Mar
  2. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Model6 Feb
  3. 2023PaLI-3 Vision Language Models: Smaller, Faster, Stronger13 Oct
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  6. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 127

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023Mistral 7Bcited by 3,065
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  13. 2019Towards VQA Models That Can Readcited by 1,696
  14. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  15. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  16. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  17. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  18. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  19. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  20. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  21. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  22. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  23. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  24. 2023LIMA: Less Is More for Alignmentcited by 1,080
  25. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  26. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  27. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  28. 2023Vision Transformers Need Registerscited by 844
  29. 2016A Diagram Is Worth A Dozen Imagescited by 793
  30. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  31. 2021Going deeper with Image Transformerscited by 759
  32. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  33. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  34. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  35. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  36. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  37. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  38. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692

50 most cited of 127 All 127 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 110; 2025: 149; 2026: 62.

321 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  4. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  5. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  6. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  7. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  8. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  9. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  10. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  11. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  12. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
38 more
  1. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
  2. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  3. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  4. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  5. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  6. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  7. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  8. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  9. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136
  10. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  11. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  12. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  13. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  14. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 110
  15. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  16. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 101
  17. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  18. 2024Merlin: A Computed Tomography Vision-Language Foundation Model and Datasetcited by 97
  19. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  20. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  21. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 87
  22. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  23. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  24. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  25. 2024Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Comparecited by 66
  26. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  27. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61
  28. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 58
  29. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  30. 2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 57
  31. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  32. 2024Needle In A Multimodal Haystackcited by 56
  33. 2025OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extractioncited by 54
  34. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  35. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 53
  36. 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
  37. 2025FineVision: Open Data Is All You Needcited by 51
  38. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 50

50 most cited of 321 All 321 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024An Introduction to Vision-Language Modeling
  2. 2026Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference
  3. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  4. 2025Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities
  5. 2025Hidden in plain sight: VLMs overlook their visual representations
  6. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  7. 2026Empirical Recipes for Efficient and Compact Vision-Language Models
  8. 2025Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation