mlsanityData to 7 Oct 2026

EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation

arXiv 2604.14306 ↗First posted 15 Apr 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

While Large Language Models (LLMs) have demonstrated high proficiency on English-centric medical examinations, their performance often declines when faced with non-English languages and multimodal diagnostic tasks. This study protocol describes the development of EuropeMedQA, the first comprehensive, multilingual, and multimodal medical examination dataset sourced from official regulatory exams in Italy, France, Spain, and Portugal. Following FAIR data principles and SPIRIT-AI guidelines, we describe a rigorous curation process and an automated translation pipeline for comparative analysis. We evaluate contemporary multimodal LLMs using a zero-shot, strictly constrained prompting strategy to assess cross-lingual transfer and visual reasoning. EuropeMedQA aims to provide a contamination-resistant benchmark that reflects the complexity of European clinical practices and fosters the development of more generalizable medical AI.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 7 of its 16 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

7 of 16 references.

References · 7

  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  2. 2024Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questionscited by 106
  3. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answeringcited by 43
  4. 2024Polish-English medical knowledge transfer: A new benchmark and resultscited by 8
  5. 2024RealMedQA: A pilot biomedical question answering dataset containing realistic clinical questionscited by 5
  6. 2025Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answeringcited by 3
  7. 2025Evaluating LLMs in Medicine: A Call for Rigor, Transparencycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation
  2. 2026Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQA
  3. 2025Evaluating Large Language Models on the Spanish Medical Intern Resident (MIR) Examination 2024/2025:A Comparative Analysis of Clinical Reasoning and Knowledge Application
  4. 2025MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks
  5. 2025BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text
  6. 2025PersianMedQA: Evaluating Large Language Models on a Persian-English Bilingual Medical Question Answering Benchmark
  7. 2025Multilingual Medical Reasoning for Question Answering with Large Language Models
  8. 2026Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguese