# WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation

arXiv: https://arxiv.org/abs/2410.12722 · HTML: https://mlsanity.com/paper/2410.12722/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 16 of 16)

- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [A Careful Examination of Large Language Model Performance on Grade School Arithmetic](https://mlsanity.com/paper/2405.00332/) (2024)
- [HEAD-QA: A Healthcare Dataset for Complex Reasoning](https://mlsanity.com/paper/1906.04701/) (2019)
- [Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis](https://mlsanity.com/paper/2310.09909/) (2023)
- [Unintended Impacts of LLM Alignment on Global Representation](https://mlsanity.com/paper/2402.15018/) (2024)
- [A Bibliometric Review of Large Language Models Research from 2017 to 2023](https://mlsanity.com/paper/2304.02020/) (2023)
- [Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V](https://mlsanity.com/paper/2310.19061/) (2023)
- [Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AI](https://mlsanity.com/paper/2309.12444/) (2023)
- [KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations](https://mlsanity.com/paper/2403.01469/) (2024)
- [Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks](https://mlsanity.com/paper/2406.12066/) (2024)
- [Language model developers should report train-test overlap](https://mlsanity.com/paper/2410.08385/) (2024)
- [Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias](https://mlsanity.com/paper/2405.05506/) (2024)

## Built on by (most cited first, 11 of 11)

- [Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens](https://mlsanity.com/paper/2511.19418/) (2025)
- [LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation](https://mlsanity.com/paper/2506.04078/) (2025)
- [Enterprise Large Language Model Evaluation Benchmark](https://mlsanity.com/paper/2506.20274/) (2025)
- [Diffusion Instruction Tuning](https://mlsanity.com/paper/2502.06814/) (2025)
- [The Art of Saying "Maybe": A Conformal Lens for Uncertainty Benchmarking in VLMs](https://mlsanity.com/paper/2509.13379/) (2025)
- [Unexplored flaws in multiple-choice VQA make benchmarking unreliable](https://mlsanity.com/paper/2511.22341/) (2025)
- [KorMedMCQA-V: A Multimodal Benchmark for Evaluating Vision-Language Models on the Korean Medical Licensing Examination](https://mlsanity.com/paper/2602.13650/) (2026)
- [M3 QuestionIng: Multi-modal Multi-span Medical Question Answering](https://mlsanity.com/paper/2606.28329/) (2026)
- [IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations](https://mlsanity.com/paper/2606.28556/) (2026)
- [Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQA](https://mlsanity.com/paper/2608.22363/) (2026)
- [Perspectives on Cross-Lingual Consistency in LLMs for Medical Questions](https://mlsanity.com/paper/2609.07687/) (2026)

## Abstract (arXiv)

Multimodal/vision language models (VLMs) are increasingly being deployed in healthcare settings worldwide, necessitating robust benchmarks to ensure their safety, efficacy, and fairness. Multiple-choice question and answer (QA) datasets derived from national medical examinations have long served as valuable evaluation tools, but existing datasets are largely text-only and available in a limited subset of languages and countries. To address these challenges, we present WorldMedQA-V, an updated multilingual, multimodal benchmarking dataset designed to evaluate VLMs in healthcare. WorldMedQA-V includes 568 labeled multiple-choice QAs paired with 568 medical images from four countries (Brazil, Israel, Japan, and Spain), covering original languages and validated English translations by native clinicians, respectively. Baseline performance for common open- and closed-source models are provided in the local language and English translations, and with and without images provided to the model. The WorldMedQA-V benchmark aims to better match AI systems to the diverse healthcare environments in which they are deployed, fostering more equitable, effective, and representative applications.