mlsanityData to 7 Oct 2026

Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation

arXiv 2501.03225 ↗First posted 6 Jan 2025cs.CV cs.AI cs.CLcited by 28

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The rapid development of vision language models (VLMs) demands rigorous and reliable evaluation. However, current visual question answering (VQA) benchmarks often depend on open-ended questions, making accurate evaluation difficult due to the variability in natural language responses. To address this, we introduce AutoConverter, an agentic framework that automatically converts these open-ended questions into multiple-choice format, enabling objective evaluation while reducing the costly multiple-choice question creation process. Our experiments demonstrate that AutoConverter can generate correct and challenging multiple-choice questions, with VLMs demonstrating consistently similar or lower accuracy on these questions compared to human-created ones. Using AutoConverter, we construct VMCBench, a benchmark created by transforming 20 existing VQA datasets into a unified multiple-choice format, totaling 9,018 questions. We comprehensively evaluate 33 state-of-the-art VLMs on VMCBench, setting a new standard for scalable, consistent, and reproducible VLM evaluation.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 64 references that are in the map. First posted on arXiv 6 Jan 2025.

Builds on

20 of 64 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024VHELM: A Holistic Evaluation of Vision Language Models9 Oct
  2. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models25 Sep
  3. 2024CogVLM2: Visual Language Models for Image and Video Understanding29 Aug
  4. 2024Building and better understanding vision-language models: insights and future directions22 Aug
  5. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  6. 2024What matters when building vision-language models?3 May
  7. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb

Other references · 13

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  6. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  7. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  9. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  10. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  11. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  12. 2024TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domainscited by 68
1 more
  1. 2024Enhancing Distractor Generation for Multiple-Choice Questions with Retrieval Augmented Pretraining and Knowledge Graph Integrationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 13.

28 papers.

Sort:

By area:

  1. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 84
  2. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 37
  3. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 30
  4. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?cited by 13
  5. 2025Data or Language Supervision: What Makes CLIP Better than DINO?cited by 13
  6. 2025NegVQA: Can Vision Language Models Understand Negation?cited by 12
  7. 2026Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decodingcited by 11
  8. 2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Modelscited by 9
  9. 2025MemLoRA: Distilling Expert Adapters for On-Device Memory Systemscited by 7
  10. 2026Qworld: Question-Specific Evaluation Criteria for LLMscited by 6
  11. 2025OASIS: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QAcited by 5
  12. 2026Large Vision-Language Models Get Lost in Attentioncited by 5
16 more
  1. 2025Bridging the Gap in Vision Language Models in Identifying Unsafe Concepts Across Modalitiescited by 4
  2. 2025Reading Between the Timelines: RAG for Answering Diachronic Questionscited by 4
  3. 2025PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistenciescited by 4
  4. 2025AutoDrive-QA: A Multiple-Choice Benchmark for Vision-Language Evaluation in Urban Autonomous Drivingcited by 2
  5. 2025Beyond Multiple Choice: Verifiable OpenQA for Robust Vision-Language RFTcited by 1
  6. 2026Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimizationcited by 1
  7. 2026Reducing Text Bias in Synthetically Generated MCQAs for VLMs in Autonomous Drivingcited by 1
  8. 2026Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Designcited by 1
  9. 2026Cognitive Pivot Points and Visual Anchoring: Unveiling and Rectifying Hallucinations in Multimodal Reasoning Modelscited by 1
  10. 2025NativQA Framework: Enabling LLMs and VLMs with Native, Local, and Everyday Knowledgecited by 0
  11. 2026Specificity-aware reinforcement learning for fine-grained open-world classificationcited by 0
  12. 2026BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QAcited by 0
  13. 2026Timage: A Generative Text-in-Image Paradigm for Fine-Tuning Vision-Language Modelscited by 0
  14. 2026C3-Bench: A Context-Aware Change Captioning Benchmarkcited by 0
  15. 2026SABRE: Scalable and Automated Benchmarking of VLMs under Stresscited by 0
  16. 2026MVVBench: Benchmarking 4D Reasoning in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types
  2. 2026AutoVQA-G: Self-Improving Agentic Framework for Automated Visual Question Answering and Grounding Annotation
  3. 2026Self-Evolving Visual Questioner
  4. 2024Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluation
  5. 2026Evaluating VQA in Vision Language Models using Cooperative Principles
  6. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?
  7. 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering
  8. 2025VQ-VA World: Towards High-Quality Visual Question-Visual Answering