mlsanityData to 7 Oct 2026

What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases

arXiv 2404.02415 ↗First posted 3 Apr 2024cs.CVcited by 13

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language (VL) models, pretrained on colossal image-text datasets, have attained broad VL competence that is difficult to evaluate. A common belief is that a small number of VL skills underlie the variety of VL tests. In this paper, we perform a large-scale transfer learning experiment aimed at discovering latent VL skills from data. We reveal interesting characteristics that have important implications for test suite design. First, generation tasks suffer from a length bias, suggesting benchmarks should balance tasks with varying output lengths. Second, we demonstrate that factor analysis successfully identifies reasonable yet surprising VL skill factors, suggesting benchmarks could leverage similar analyses for task selection. Finally, we present a new dataset, OLIVE (https://github.com/jq-zh/olive-dataset), which simulates user instructions in the wild and presents challenges dissimilar to all datasets we tested. Our findings contribute to the design of balanced and broad-coverage vision-language evaluation methods.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 65 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

42 of 65 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks4 Oct
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2019Towards VQA Models That Can Readcited by 1,696
27 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  6. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  7. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  8. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  10. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  11. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  12. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  13. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  14. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 218
  15. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  16. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  17. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  18. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  19. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  20. 2019Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learningcited by 86
  21. 2022OpenCQA: Open-ended Question Answering with Chartscited by 60
  22. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  23. 2023TouchStone: Evaluating Vision-Language Models by Language Modelscited by 54
  24. 2020Scale-Localized Abstract Reasoningcited by 45
  25. 2019The Information Complexity of Learning Tasks, their Structure and their Distancecited by 35
  26. 2021Nice perfume. How long did you marinate in it? Multimodal Sarcasm Explanationcited by 25
  27. 2021GEM: A General Evaluation Benchmark for Multimodal Taskscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 4; 2026: 1.

13 papers.

Sort:

By area:

  1. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  2. 2024SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluationcited by 31
  3. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Modelscited by 24
  4. 2025CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Mergingcited by 22
  5. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understandingcited by 17
  6. 2025Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflictscited by 11
  7. 2024Natural Language Inference Improves Compositionality in Vision-Language Modelscited by 8
  8. 2024Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMscited by 4
  9. 2024HateSieve: A Contrastive Learning Framework for Detecting and Segmenting Hateful Content in Multimodal Memescited by 2
  10. 2025Understanding Task Transfer in Vision-Language Modelscited by 2
  11. 2024A Training Data Recipe to Accelerate A* Search with Language Modelscited by 1
  12. 2025T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMscited by 1
1 more
  1. 2026Geometric Similarity in VLM Low-Level Vision Representationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
  2. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
  3. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models
  4. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  5. 2024VHELM: A Holistic Evaluation of Vision Language Models
  6. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  7. 2026On Test-Time Scaling for Vision-Language Models
  8. 2024Rethinking Overlooked Aspects in Vision-Language Models