Overview of the MedHopQA track at BioCreative IX: track description, participation and evaluation of systems for multi-hop medical question answering
Not on the map. Why?
Abstract (arXiv)
Multi-hop question answering (QA) remains a significant challenge in the biomedical domain, requiring systems to integrate information across multiple sources to answer complex questions. To address this problem, the BioCreative IX MedHopQA shared task was designed to benchmark in multi-hop reasoning for large language models (LLMs). We developed a novel dataset of 1,000 challenging QA pairs spanning diseases, genes, and chemicals, with particular emphasis on rare diseases. Each question was constructed to require two-hop reasoning through the integration of information from two distinct Wikipedia pages. The challenge attracted 48 submissions from 13 teams. Systems were evaluated using both surface string comparison and conceptual accuracy (MedCPT score). The results showed a substantial performance gap between baseline LLMs and enhanced systems. The top-ranked submission achieved an 89.30% F1 score on the MedCPT metric and an 87.30% exact match (EM) score, compared with 67.40% and 60.20%, respectively, for the zero-shot baseline. A central finding of the challenge was that retrieval-augmented generation (RAG) and related retrieval-based strategies were critical for strong performance. In addition, concept-level evaluation improved answer assessment when correct responses differed in surface form. The MedHopQA dataset is publicly available to support continued progress in this important area. Challenge materials: https://www.ncbi.nlm.nih.gov/research/bionlp/medhopqa and benchmark https://www.codabench.org/competitions/7609/
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 41 references that are in the map. First posted on arXiv 12 May 2026.
Builds on
14 of 41 references.
References · 14
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2022Large Language Models Encode Clinical Knowledgecited by 1,807
- 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
- 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
- 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
- 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 182
- 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 134
- 2024Med42-v2: A Suite of Clinical LLMscited by 89
- 2024BM25S: Orders of magnitude faster lexical search via eager sparse scoringcited by 87
- 2024Rationale-Guided Retrieval Augmented Generation for Medical Question Answeringcited by 41
- 2025DeepRAG: Integrating Hierarchical Reasoning and Process Supervision for Biomedical Multi-Hop QAcited by 2
2 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering
- 2026UETQuintet at BioCreative IX -- MedHopQA: Enhancing Biomedical QA with Selective Multi-hop Reasoning and Contextual Retrieval
- 2025BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain
- 2025Harnessing Collective Intelligence of LLMs for Robust Biomedical QA: A Multi-Model Approach
- 2025RAG-BioQA: A Retrieval-Augmented Generation Framework for Long-Form Biomedical Question Answering
- 2025Beyond Retrieval: Ensembling Cross-Encoders and GPT Rerankers with LLMs for Biomedical QA
- 2026Weaving Multi-Source Evidence for Biomedical Reasoning: The BioMedHop Benchmark and BioWeave Framework
- 2024LLM-MedQA: Enhancing Medical Question Answering through Case Studies in Large Language Models