mlsanityData to 14 Oct 2026

WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation

arXiv 2410.12722 ↗First posted 16 Oct 2024cs.CLcited by 11

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Multimodal/vision language models (VLMs) are increasingly being deployed in healthcare settings worldwide, necessitating robust benchmarks to ensure their safety, efficacy, and fairness. Multiple-choice question and answer (QA) datasets derived from national medical examinations have long served as valuable evaluation tools, but existing datasets are largely text-only and available in a limited subset of languages and countries. To address these challenges, we present WorldMedQA-V, an updated multilingual, multimodal benchmarking dataset designed to evaluate VLMs in healthcare. WorldMedQA-V includes 568 labeled multiple-choice QAs paired with 568 medical images from four countries (Brazil, Israel, Japan, and Spain), covering original languages and validated English translations by native clinicians, respectively. Baseline performance for common open- and closed-source models are provided in the local language and English translations, and with and without images provided to the model. The WorldMedQA-V benchmark aims to better match AI systems to the diverse healthcare environments in which they are deployed, fostering more equitable, effective, and representative applications.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 38 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

16 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V29 Oct

Other references · 15

  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  2. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  3. 2021Ethical and social risks of harm from Language Modelscited by 966
  4. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  5. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  6. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  7. 2019HEAD-QA: A Healthcare Dataset for Complex Reasoningcited by 101
  8. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  9. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  10. 2023A Bibliometric Review of Large Language Models Research from 2017 to 2023cited by 57
  11. 2023Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AIcited by 32
  12. 2024KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinationscited by 22
3 more
  1. 2024Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarkscited by 17
  2. 2024Language model developers should report train-test overlapcited by 16
  3. 2024Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Biascited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokenscited by 71
  2. 2025LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validationcited by 34
  3. 2025Enterprise Large Language Model Evaluation Benchmarkcited by 5
  4. 2025Diffusion Instruction Tuningcited by 2
  5. 2025The Art of Saying "Maybe": A Conformal Lens for Uncertainty Benchmarking in VLMscited by 2
  6. 2025Unexplored flaws in multiple-choice VQA make benchmarking unreliablecited by 1
  7. 2026KorMedMCQA-V: A Multimodal Benchmark for Evaluating Vision-Language Models on the Korean Medical Licensing Examinationcited by 1
  8. 2026M3 QuestionIng: Multi-modal Multi-span Medical Question Answeringcited by 0
  9. 2026IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversationscited by 0
  10. 2026Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQAcited by 0
  11. 2026Perspectives on Cross-Lingual Consistency in LLMs for Medical Questionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation
  2. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM
  3. 2024FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Tasks
  4. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering
  5. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
  6. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark
  7. 2024SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Tasks
  8. 2025PersianMedQA: Evaluating Large Language Models on a Persian-English Bilingual Medical Question Answering Benchmark