mlsanityData to 7 Oct 2026

MedExQA: Medical Question Answering Benchmark with Multiple Explanations

arXiv 2406.06331 ↗First posted 10 Jun 2024cs.CL cs.AIcited by 58

Not on the map. Why?

Abstract (arXiv)

This paper introduces MedExQA, a novel benchmark in medical question-answering, to evaluate large language models' (LLMs) understanding of medical knowledge through explanations. By constructing datasets across five distinct medical specialties that are underrepresented in current datasets and further incorporating multiple explanations for each question-answer pair, we address a major gap in current medical QA benchmarks which is the absence of comprehensive assessments of LLMs' ability to generate nuanced medical explanations. Our work highlights the importance of explainability in medical LLMs, proposes an effective methodology for evaluating models beyond classification accuracy, and sheds light on one specific domain, speech language pathology, where current LLMs including GPT4 lack good understanding. Our results show generation evaluation with multiple explanations aligns better with human assessment, highlighting an opportunity for a more robust automated comprehension assessment for LLMs. To diversify open-source medical LLMs (currently mostly based on Llama2), this work also proposes a new medical model, MedPhi-2, based on Phi-2 (2.7B). The model outperformed medical LLMs based on Llama2-70B in generating explanations, showing its effectiveness in the resource-constrained medical domain. We will share our benchmark datasets and the trained model.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 11 of its 23 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

11 of 23 references.

References · 11

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  7. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  8. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  9. 2023AlpaCare:Instruction-tuned Large Language Models for Medical Applicationcited by 88
  10. 2023Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notescited by 51
  11. 2023Benchmarking and Analyzing In-context Learning, Fine-tuning and Supervised Learning for Biomedical Knowledge Curation: a focused study on chemical entities of biological interestcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 32; 2026: 24.

58 papers.

Sort:

By area:

  1. 2025MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworkscited by 38
  2. 2025A Comprehensive Survey on the Trustworthiness of Large Language Models in Healthcarecited by 27
  3. 2024A Benchmark for Long-Form Medical Question Answeringcited by 25
  4. 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Modelscited by 21
  5. 2025CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answeringcited by 20
  6. 20253MDBench: Medical Multimodal Multi-agent Dialogue Benchmarkcited by 16
  7. 2025AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Modelscited by 16
  8. 2025A Case Study Exploring the Current Landscape of Synthetic Medical Record Generation with Commercial LLMscited by 13
  9. 2025BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domaincited by 10
  10. 2025MIRIAD: Augmenting LLMs with millions of medical query-response pairscited by 8
  11. 2026MAWARITH: A Dataset and Benchmark for Legal Inheritance Reasoning with LLMscited by 8
  12. 2025Trustworthy Medical Question Answering: An Evaluation-Centric Surveycited by 7
38 more
  1. 2025Applications of Large Models in Medicinecited by 6
  2. 2026GSEM: Graph-based Self-Evolving Memory for Experience Augmented Clinical Reasoningcited by 5
  3. 2025TCM-3CEval: A Triaxial Benchmark for Assessing Responses from Large Language Models in Traditional Chinese Medicinecited by 4
  4. 2025Beyond Memorization: A Rigorous Evaluation Framework for Medical Knowledge Editingcited by 4
  5. 2025MSCoRe: A Benchmark for Multi-Stage Collaborative Reasoning in LLM Agentscited by 3
  6. 2025CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Researchcited by 3
  7. 2026Patient-Similarity Cohort Reasoning in Clinical Text-to-SQLcited by 3
  8. 2025From Retrieval to Generation: Comparing Different Approachescited by 2
  9. 2025Evaluating Answer Reranking Strategies in Time-sensitive Question Answeringcited by 2
  10. 2025TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verificationcited by 2
  11. 2025Silence is Not Consensus: Disrupting Agreement Bias in Multi-Agent LLMs via Catfish Agent for Clinical Decision Makingcited by 2
  12. 2025High Accuracy, Less Talk (HALT): Reliable LLMs through Capability-Aligned Finetuningcited by 2
  13. 2025Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligencecited by 2
  14. 2025Benchmarking for Domain-Specific LLMs: A Case Study on Academia and Beyondcited by 2
  15. 2026Mind the Ambiguity: Aleatoric Uncertainty Quantification in LLMs for Safe Medical Question Answeringcited by 2
  16. 2026MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answeringcited by 2
  17. 2026Large Language Models Lack Temporal Awareness of Medical Knowledgecited by 2
  18. 2026EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMscited by 2
  19. 2025DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domaincited by 1
  20. 2025Continually Self-Improving Language Models for Bariatric Surgery Question--Answeringcited by 1
  21. 2025TeleEval-OS: Performance evaluations of large language models for operations schedulingcited by 1
  22. 2025MedBench-IT: A Comprehensive Benchmark for Evaluating Large Language Models on Italian Medical Entrance Examinationscited by 1
  23. 2025Filling in the Clinical Gaps in Benchmark: Case for HealthBench for the Japanese medical systemcited by 1
  24. 2025Risk Profiling and Modulation for LLMscited by 1
  25. 2026MLB: A Scenario-Driven Benchmark for Evaluating Large Language Models in Clinical Applicationscited by 1
  26. 2026Medical Context Distorts Decisions in Clinical Vision Language Modelscited by 1
  27. 2024Evidence Is All You Need: Ordering Imaging Studies via Language Model Alignment with the ACR Appropriateness Criteriacited by 0
  28. 2025IHC-LLMiner: Automated extraction of tumour immunohistochemical profiles from PubMed abstracts using large language modelscited by 0
  29. 2025NurValues: Real-World Nursing Values Evaluation for Large Language Models in Clinical Contextcited by 0
  30. 2025PerMedCQA: Benchmarking Large Language Models on Medical Consumer Question Answering in Persian Languagecited by 0
  31. 2025Safer in Translation? Presupposition Robustness in Indic Languagescited by 0
  32. 2026Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Makingcited by 0
  33. 2026Quantifying Hallucinations in Language Language Models on Medical Textbookscited by 0
  34. 2026From Exposure to Internalization: Dual-Stream Calibration for In-context Clinical Reasoningcited by 0
  35. 2026AROMA: Augmented Reasoning Over a Multimodal Architecture for Virtual Cell Genetic Perturbation Modelingcited by 0
  36. 2026VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphscited by 0
  37. 2026MedMeta: A Benchmark for LLMs in Synthesizing Meta-Analysis Conclusion from Medical Studiescited by 0
  38. 2026How Good LLMs Are at Answering Bangla Medical Visual Questions? Dataset and Benchmarkingcited by 0

50 most cited of 58 All 58 (JSON)