# MediQAl: A French Medical Question Answering Dataset for Knowledge and Reasoning Evaluation

arXiv: https://arxiv.org/abs/2507.20917 · HTML: https://mlsanity.com/paper/2507.20917/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 28 Jul 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 15 of 15)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs](https://mlsanity.com/paper/2412.18925/) (2024)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding](https://mlsanity.com/paper/2501.18362/) (2025)
- [UltraMedical: Building Specialized Generalists in Biomedicine](https://mlsanity.com/paper/2406.03949/) (2024)
- [Few shot chain-of-thought driven reasoning to prompt LLMs for open ended medical question answering](https://mlsanity.com/paper/2403.04890/) (2024)
- [FineMedLM-o1: Enhancing Medical Knowledge Reasoning Ability of LLM from Supervised Fine-Tuning to Test-Time Training](https://mlsanity.com/paper/2501.09213/) (2025)
- [Large Language Model Benchmarks in Medical Tasks](https://mlsanity.com/paper/2410.21348/) (2024)
- [Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings](https://mlsanity.com/paper/2410.13153/) (2024)

## Built on by (most cited first, 5 of 5)

- [Who Judges the Judge? Evaluating LLM-as-a-Judge for French Medical open-ended QA](https://mlsanity.com/paper/2603.04033/) (2026)
- [Evaluating Large Language Model Raters for German Open-Response Clinical Questions: A Physician-Annotated Benchmark Study of Agreement, Evaluator Bias, and Abstention](https://mlsanity.com/paper/2607.01103/) (2026)
- [MedInjection-FR: Exploring the Role of Native, Synthetic, and Translated Data in Biomedical Instruction Tuning](https://mlsanity.com/paper/2603.06905/) (2026)
- [Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA](https://mlsanity.com/paper/2606.19266/) (2026)
- [EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding](https://mlsanity.com/paper/2609.32352/) (2026)

## Abstract (arXiv)

This work introduces MediQAl, a French medical question answering dataset designed to evaluate the capabilities of language models in factual medical recall and reasoning over real-world clinical scenarios. MediQAl contains 32,603 questions sourced from French medical examinations across 41 medical subjects. The dataset includes three tasks: (i) Multiple-Choice Question with Unique answer, (ii) Multiple-Choice Question with Multiple answer, and (iii) Open-Ended Question with Short-Answer. Each question is labeled as Understanding or Reasoning, enabling a detailed analysis of models' cognitive capabilities. We validate the MediQAl dataset through extensive evaluation with 14 large language models, including recent reasoning-augmented models, and observe a significant performance gap between factual recall and reasoning tasks. Our evaluation provides a comprehensive benchmark for assessing language models' performance on French medical question answering, addressing a crucial gap in multilingual resources for the medical domain.