# EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation

arXiv: https://arxiv.org/abs/2604.14306 · HTML: https://mlsanity.com/paper/2604.14306/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 16 references that are in the map.
- First posted on arXiv 15 Apr 2026.

## Builds on (most cited first, 7 of 7)

- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions](https://mlsanity.com/paper/2402.18060/) (2024)
- [MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering](https://mlsanity.com/paper/2404.05590/) (2024)
- [Polish-English medical knowledge transfer: A new benchmark and results](https://mlsanity.com/paper/2412.00559/) (2024)
- [RealMedQA: A pilot biomedical question answering dataset containing realistic clinical questions](https://mlsanity.com/paper/2408.08624/) (2024)
- [Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering](https://mlsanity.com/paper/2511.19220/) (2025)
- [Evaluating LLMs in Medicine: A Call for Rigor, Transparency](https://mlsanity.com/paper/2507.08916/) (2025)

## Abstract (arXiv)

While Large Language Models (LLMs) have demonstrated high proficiency on English-centric medical examinations, their performance often declines when faced with non-English languages and multimodal diagnostic tasks. This study protocol describes the development of EuropeMedQA, the first comprehensive, multilingual, and multimodal medical examination dataset sourced from official regulatory exams in Italy, France, Spain, and Portugal. Following FAIR data principles and SPIRIT-AI guidelines, we describe a rigorous curation process and an automated translation pipeline for comparative analysis. We evaluate contemporary multimodal LLMs using a zero-shot, strictly constrained prompting strategy to assess cross-lingual transfer and visual reasoning. EuropeMedQA aims to provide a contamination-resistant benchmark that reflects the complexity of European clinical practices and fosters the development of more generalizable medical AI.