# BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain

arXiv: https://arxiv.org/abs/2505.22240 · HTML: https://mlsanity.com/paper/2505.22240/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 17 of 17)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs](https://mlsanity.com/paper/2412.18925/) (2024)
- [Do Large Language Models Latently Perform Multi-Hop Reasoning?](https://mlsanity.com/paper/2402.16837/) (2024)
- [Multi-Step Reasoning with Large Language Models, a Survey](https://mlsanity.com/paper/2407.11511/) (2024)
- [Can Language Models be Biomedical Knowledge Bases?](https://mlsanity.com/paper/2109.07154/) (2021)
- [UltraMedical: Building Specialized Generalists in Biomedicine](https://mlsanity.com/paper/2406.03949/) (2024)
- [MedExQA: Medical Question Answering Benchmark with Multiple Explanations](https://mlsanity.com/paper/2406.06331/) (2024)
- [MoreHopQA: More Than Multi-hop Reasoning](https://mlsanity.com/paper/2406.13397/) (2024)
- [KGARevion: An AI Agent for Knowledge-Intensive Biomedical QA](https://mlsanity.com/paper/2410.04660/) (2024)
- [BioLORD-2023: Semantic Textual Representations Fusing LLM and Clinical Knowledge Graph Insights](https://mlsanity.com/paper/2311.16075/) (2023)
- [Triggering Multi-Hop Reasoning for Question Answering in Language Models using Soft Prompts and Random Walks](https://mlsanity.com/paper/2306.04009/) (2023)
- [Biomedical Multi-hop Question Answering Using Knowledge Graph Embeddings and Language Models](https://mlsanity.com/paper/2211.05351/) (2022)

## Built on by (most cited first, 10 of 10)

- [DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections](https://mlsanity.com/paper/2508.15851/) (2025)
- [PaperSearchQA: Learning to Search and Reason over Scientific Papers with RLVR](https://mlsanity.com/paper/2601.18207/) (2026)
- [BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence](https://mlsanity.com/paper/2603.07931/) (2026)
- [VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observations](https://mlsanity.com/paper/2603.16506/) (2026)
- [MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering](https://mlsanity.com/paper/2605.12361/) (2026)
- [Kinship Data Benchmark for Multi-hop Reasoning](https://mlsanity.com/paper/2601.07794/) (2026)
- [RareAgent: Self-Evolving Reasoning for Drug Repurposing in Rare Diseases](https://mlsanity.com/paper/2510.05764/) (2025)
- [Condition-Gated Reasoning for Context-Dependent Biomedical Question Answering](https://mlsanity.com/paper/2602.17911/) (2026)
- [AMNESIA: A Large Scale Medical Unlearning Benchmark Suite with Disease-Informed Analysis](https://mlsanity.com/paper/2605.30599/) (2026)
- [SSE-Bio: A Structured Self-Evolving Agent with Agentic Retrieval Policy for Multi-Hop Biomedical Reasoning](https://mlsanity.com/paper/2608.22132/) (2026)

## Abstract (arXiv)

Biomedical reasoning often requires traversing interconnected relationships across entities such as drugs, diseases, and proteins. Despite the increasing prominence of large language models (LLMs), existing benchmarks lack the ability to evaluate multi-hop reasoning in the biomedical domain, particularly for queries involving one-to-many and many-to-many relationships. This gap leaves the critical challenges of biomedical multi-hop reasoning underexplored. To address this, we introduce BioHopR, a novel benchmark designed to evaluate multi-hop, multi-answer reasoning in structured biomedical knowledge graphs. Built from the comprehensive PrimeKG, BioHopR includes 1-hop and 2-hop reasoning tasks that reflect real-world biomedical complexities. Evaluations of state-of-the-art models reveal that O3-mini, a proprietary reasoning-focused model, achieves 37.93% precision on 1-hop tasks and 14.57% on 2-hop tasks, outperforming proprietary models such as GPT4O and open-source biomedical models including HuatuoGPT-o1-70B and Llama-3.3-70B. However, all models exhibit significant declines in multi-hop performance, underscoring the challenges of resolving implicit reasoning steps in the biomedical domain. By addressing the lack of benchmarks for multi-hop reasoning in biomedical domain, BioHopR sets a new standard for evaluating reasoning capabilities and highlights critical gaps between proprietary and open-source models while paving the way for future advancements in biomedical LLMs.