# MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering

arXiv: https://arxiv.org/abs/2605.12361 · HTML: https://mlsanity.com/paper/2605.12361/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 12 May 2026.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 20 of 20)

- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [Constructing Datasets for Multi-hop Reading Comprehension Across Documents](https://mlsanity.com/paper/1710.06481/) (2017)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [Time Travel in LLMs: Tracing Data Contamination in Large Language Models](https://mlsanity.com/paper/2308.08493/) (2023)
- [Lessons from Natural Language Inference in the Clinical Domain](https://mlsanity.com/paper/1808.06752/) (2018)
- [emrQA: A Large Corpus for Question Answering on Electronic Medical Records](https://mlsanity.com/paper/1809.00732/) (2018)
- [MedExQA: Medical Question Answering Benchmark with Multiple Explanations](https://mlsanity.com/paper/2406.06331/) (2024)
- [LongHealth: A Question Answering Benchmark with Long Clinical Documents](https://mlsanity.com/paper/2401.14490/) (2024)
- [K-QA: A Real-World Medical Q&A Benchmark](https://mlsanity.com/paper/2401.14493/) (2024)
- [AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset](https://mlsanity.com/paper/2411.15640/) (2024)
- [BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain](https://mlsanity.com/paper/2505.22240/) (2025)
- [MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering](https://mlsanity.com/paper/2406.05845/) (2024)
- [LLMs Outperform Experts on Challenging Biology Benchmarks](https://mlsanity.com/paper/2505.06108/) (2025)
- [RealMedQA: A pilot biomedical question answering dataset containing realistic clinical questions](https://mlsanity.com/paper/2408.08624/) (2024)
- [Overview of the ClinIQLink 2025 Shared Task on Medical Question-Answering](https://mlsanity.com/paper/2506.21597/) (2025)
- [A Large-Scale Benchmark for Evaluating Large Language Models on Medical Question Answering in Romanian](https://mlsanity.com/paper/2508.16390/) (2025)

## Built on by (most cited first, 2 of 2)

- [Weaving Multi-Source Evidence for Biomedical Reasoning: The BioMedHop Benchmark and BioWeave Framework](https://mlsanity.com/paper/2606.16211/) (2026)
- [Repair Before Reinforce: Context-Augmented Knowledge Graph Reasoning for Multi-Hop Question Answering](https://mlsanity.com/paper/2609.12230/) (2026)

## Abstract (arXiv)

Evaluating large language models (LLMs) in the biomedical domain requires benchmarks that can distinguish reasoning from pattern matching and remain discriminative as model capabilities improve. Existing biomedical question answering (QA) benchmarks are limited in this respect. Multiple-choice formats can allow models to succeed through answer elimination rather than inference, while widely circulated exam-style datasets are increasingly vulnerable to performance saturation and training data contamination. Multi-hop reasoning, defined as the ability to integrate information across multiple sources to derive an answer, is central to clinically meaningful tasks such as diagnostic support, literature-based discovery, and hypothesis generation, yet remains underrepresented in current biomedical QA benchmarks. MedHopQA is a disease-centered multi-hop reasoning benchmark consisting of 1,000 expert-curated question-answer pairs introduced as a shared task at BioCreative IX. Each question requires synthesis of information across two distinct Wikipedia articles, and answers are provided in an open-ended free-text format. Gold annotations are augmented with ontology-grounded synonym sets from MONDO, NCBI Gene, and NCBI Taxonomy to support both lexical and concept-level evaluation. MedHopQA was constructed through a structured process combining human annotation, triage, iterative verification, and LLM-as-a-judge validation. To reduce leaderboard gaming and contamination risk, the 1,000 scored questions are embedded within a publicly downloadable set of 10,000 questions, with answers withheld, on a CodaBench leaderboard. MedHopQA provides both a benchmark and a reusable framework for constructing future biomedical QA datasets that prioritize compositional reasoning, saturation resistance, and contamination resistance as core design constraints.