mlsanityData to 7 Oct 2026

Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy

arXiv 2402.07270 ↗First posted 11 Feb 2024cs.CV cs.CL cs.LGcited by 26

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The evaluation of text-generative vision-language models is a challenging yet crucial endeavor. By addressing the limitations of existing Visual Question Answering (VQA) benchmarks and proposing innovative evaluation methodologies, our research seeks to advance our understanding of these models' capabilities. We propose a novel VQA benchmark based on well-known visual classification datasets which allows a granular evaluation of text-generative vision-language models and their comparison with discriminative vision-language models. To improve the assessment of coarse answers on fine-grained classification tasks, we suggest using the semantic hierarchy of the label space to ask automatically generated follow-up questions about the ground-truth category. Finally, we compare traditional NLP and LLM-based metrics for the problem of evaluating model predictions given ground-truth answers. We perform a human evaluation study upon which we base our decision on the final metric. We apply our benchmark to a suite of vision-language models and show a detailed comparison of their abilities on object, action, and attribute classification. Our contributions aim to lay the foundation for more precise and meaningful assessments, facilitating targeted progress in the exciting field of vision-language modeling.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 52 references that are in the map. First posted on arXiv 11 Feb 2024.

Builds on

39 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks22 Nov
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
24 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  8. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  9. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  10. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  11. 2016Visual Dialogcited by 377
  12. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  13. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  14. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  15. 2021Learning Compact Metrics for MTcited by 93
  16. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  17. 2023TouchStone: Evaluating Vision-Language Models by Language Modelscited by 54
  18. 2018Learning to Evaluate Image Captioningcited by 53
  19. 2021Semantic Answer Similarity for Evaluating Question Answering Modelscited by 47
  20. 2020MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metricscited by 42
  21. 2023Vocabulary-free Image Classificationcited by 35
  22. 2022Open-vocabulary Attribute Detectioncited by 28
  23. 2019Challenges and Prospects in Vision and Language Researchcited by 24
  24. 2021GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 12; 2026: 8.

26 papers.

Sort:

By area:

  1. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  2. 2024VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledgecited by 50
  3. 2024The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?cited by 24
  4. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understandingcited by 17
  5. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  6. 2024Towards Foundation Models for 3D Vision: How Close Are We?cited by 13
  7. 2025Reasoning-Enhanced Domain-Adaptive Pretraining of Multimodal Large Language Models for Short Video Content Governancecited by 13
  8. 2025Taxonomy-Aware Evaluation of Vision-Language Modelscited by 11
  9. 2025Directional Gradient Projection for Robust Fine-Tuning of Foundation Modelscited by 10
  10. 2024Video Question Answering for People with Visual Impairments Using an Egocentric 360-Degree Cameracited by 9
  11. 2025Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generationcited by 6
  12. 2024A Hitchhikers Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoningcited by 4
14 more
  1. 2025VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videoscited by 4
  2. 2025The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation from Recognition to Reasoningcited by 4
  3. 2025EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoningcited by 4
  4. 2026VRIQ: Benchmarking and Analyzing Visual-Reasoning IQ of VLMscited by 4
  5. 2026Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Modelscited by 4
  6. 2026MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documentscited by 3
  7. 2025Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Modelscited by 2
  8. 2025Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimizationcited by 1
  9. 2026FaithSCAN: Model-Driven Single-Pass Hallucination Detection for Faithful Visual Question Answeringcited by 1
  10. 2025Patent Figure Classification using Large Vision-language Modelscited by 0
  11. 2026FuzzingRL: Reinforcement Fuzz-Testing for Revealing VLM Failurescited by 0
  12. 2026Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiologycited by 0
  13. 2026What Does Your Short-Answer VQA Score Actually Measure? Evaluator-Dependent Instability in Multimodal Short-Answer Benchmarkscited by 0
  14. 2026EgoErrorVQA: Assess Egocentric Comprehension Capabilities through Procedural Errors for Ego-Agentic AIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Language
  2. 2026Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models
  3. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types
  4. 2024VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation
  5. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?
  6. 2025When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs
  7. 2024Towards Flexible Evaluation for Generative Visual Question Answering
  8. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities