# MedExQA: Medical Question Answering Benchmark with Multiple Explanations

arXiv: https://arxiv.org/abs/2406.06331 · HTML: https://mlsanity.com/paper/2406.06331/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://mlsanity.com/paper/2312.15166/) (2023)
- [AlpaCare:Instruction-tuned Large Language Models for Medical Application](https://mlsanity.com/paper/2310.14558/) (2023)
- [Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes](https://mlsanity.com/paper/2309.00237/) (2023)
- [Benchmarking and Analyzing In-context Learning, Fine-tuning and Supervised Learning for Biomedical Knowledge Curation: a focused study on chemical entities of biological interest](https://mlsanity.com/paper/2312.12989/) (2023)

## Built on by (most cited first, 30 of 60)

- [MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworks](https://mlsanity.com/paper/2503.07459/) (2025)
- [A Comprehensive Survey on the Trustworthiness of Large Language Models in Healthcare](https://mlsanity.com/paper/2502.15871/) (2025)
- [A Benchmark for Long-Form Medical Question Answering](https://mlsanity.com/paper/2411.09834/) (2024)
- [Automating Expert-Level Medical Reasoning Evaluation of Large Language Models](https://mlsanity.com/paper/2507.07988/) (2025)
- [CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering](https://mlsanity.com/paper/2506.08584/) (2025)
- [AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Models](https://mlsanity.com/paper/2505.17312/) (2025)
- [3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark](https://mlsanity.com/paper/2504.13861/) (2025)
- [A Case Study Exploring the Current Landscape of Synthetic Medical Record Generation with Commercial LLMs](https://mlsanity.com/paper/2504.14657/) (2025)
- [BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain](https://mlsanity.com/paper/2505.22240/) (2025)
- [MIRIAD: Augmenting LLMs with millions of medical query-response pairs](https://mlsanity.com/paper/2506.06091/) (2025)
- [MAWARITH: A Dataset and Benchmark for Legal Inheritance Reasoning with LLMs](https://mlsanity.com/paper/2603.07539/) (2026)
- [Trustworthy Medical Question Answering: An Evaluation-Centric Survey](https://mlsanity.com/paper/2506.03659/) (2025)
- [Applications of Large Models in Medicine](https://mlsanity.com/paper/2502.17132/) (2025)
- [GSEM: Graph-based Self-Evolving Memory for Experience Augmented Clinical Reasoning](https://mlsanity.com/paper/2603.22096/) (2026)
- [TCM-3CEval: A Triaxial Benchmark for Assessing Responses from Large Language Models in Traditional Chinese Medicine](https://mlsanity.com/paper/2503.07041/) (2025)
- [Beyond Memorization: A Rigorous Evaluation Framework for Medical Knowledge Editing](https://mlsanity.com/paper/2506.03490/) (2025)
- [MSCoRe: A Benchmark for Multi-Stage Collaborative Reasoning in LLM Agents](https://mlsanity.com/paper/2509.17628/) (2025)
- [CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Research](https://mlsanity.com/paper/2510.11985/) (2025)
- [Patient-Similarity Cohort Reasoning in Clinical Text-to-SQL](https://mlsanity.com/paper/2601.09876/) (2026)
- [From Retrieval to Generation: Comparing Different Approaches](https://mlsanity.com/paper/2502.20245/) (2025)
- [Evaluating Answer Reranking Strategies in Time-sensitive Question Answering](https://mlsanity.com/paper/2503.04972/) (2025)
- [TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification](https://mlsanity.com/paper/2505.18283/) (2025)
- [Silence is Not Consensus: Disrupting Agreement Bias in Multi-Agent LLMs via Catfish Agent for Clinical Decision Making](https://mlsanity.com/paper/2505.21503/) (2025)
- [High Accuracy, Less Talk (HALT): Reliable LLMs through Capability-Aligned Finetuning](https://mlsanity.com/paper/2506.04051/) (2025)
- [Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligence](https://mlsanity.com/paper/2507.10281/) (2025)
- [Benchmarking for Domain-Specific LLMs: A Case Study on Academia and Beyond](https://mlsanity.com/paper/2508.07353/) (2025)
- [Mind the Ambiguity: Aleatoric Uncertainty Quantification in LLMs for Safe Medical Question Answering](https://mlsanity.com/paper/2601.17284/) (2026)
- [Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks](https://mlsanity.com/paper/2605.01417/) (2026)
- [MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering](https://mlsanity.com/paper/2605.12361/) (2026)
- [Large Language Models Lack Temporal Awareness of Medical Knowledge](https://mlsanity.com/paper/2605.13045/) (2026)

## Abstract (arXiv)

This paper introduces MedExQA, a novel benchmark in medical question-answering, to evaluate large language models' (LLMs) understanding of medical knowledge through explanations. By constructing datasets across five distinct medical specialties that are underrepresented in current datasets and further incorporating multiple explanations for each question-answer pair, we address a major gap in current medical QA benchmarks which is the absence of comprehensive assessments of LLMs' ability to generate nuanced medical explanations. Our work highlights the importance of explainability in medical LLMs, proposes an effective methodology for evaluating models beyond classification accuracy, and sheds light on one specific domain, speech language pathology, where current LLMs including GPT4 lack good understanding. Our results show generation evaluation with multiple explanations aligns better with human assessment, highlighting an opportunity for a more robust automated comprehension assessment for LLMs. To diversify open-source medical LLMs (currently mostly based on Llama2), this work also proposes a new medical model, MedPhi-2, based on Phi-2 (2.7B). The model outperformed medical LLMs based on Llama2-70B in generating explanations, showing its effectiveness in the resource-constrained medical domain. We will share our benchmark datasets and the trained model.