# MedProbeBench: Systematic Benchmarking at Deep Evidence Integration for Expert-level Medical Guideline

arXiv: https://arxiv.org/abs/2604.18418 · HTML: https://mlsanity.com/paper/2604.18418/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 48 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Builds on (most cited first, 21 of 21)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Benchmarking Retrieval-Augmented Generation for Medicine](https://mlsanity.com/paper/2402.13178/) (2024)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025)
- [MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding](https://mlsanity.com/paper/2501.18362/) (2025)
- [AgentScope: A Flexible yet Robust Multi-Agent Platform](https://mlsanity.com/paper/2402.14034/) (2024)
- [ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents](https://mlsanity.com/paper/2511.07685/) (2025)
- [MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling](https://mlsanity.com/paper/2511.11793/) (2025)
- [MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow](https://mlsanity.com/paper/2503.18968/) (2025)
- [TxAgent: An AI Agent for Therapeutic Reasoning Across a Universe of Tools](https://mlsanity.com/paper/2503.10970/) (2025)
- [GMAI-VL-R1: Harnessing Reinforcement Learning for Multimodal Medical Reasoning](https://mlsanity.com/paper/2504.01886/) (2025)
- [MedBrowseComp: Benchmarking Medical Deep Research and Computer Use](https://mlsanity.com/paper/2505.14963/) (2025)
- [MedGround-R1: Advancing Medical Image Grounding via Spatial-Semantic Rewarded Group Relative Policy Optimization](https://mlsanity.com/paper/2507.02994/) (2025)
- [MedAide: Information Fusion and Anatomy of Medical Intents via LLM-based Agent Collaboration](https://mlsanity.com/paper/2410.12532/) (2024)
- [MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models](https://mlsanity.com/paper/2505.11613/) (2025)
- [MedQ-Bench: Evaluating and Exploring Medical Image Quality Assessment Abilities in MLLMs](https://mlsanity.com/paper/2510.01691/) (2025)
- [MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark](https://mlsanity.com/paper/2604.10755/) (2026)

## Built on by (most cited first, 1 of 1)

- [Workflow Closure Is Not Scientific Closure in Auto-Research Systems](https://mlsanity.com/paper/2605.26200/) (2026)

## Abstract (arXiv)

Recent advances in deep research systems enable large language models to retrieve, synthesize, and reason over large-scale external knowledge. In medicine, developing clinical guidelines critically depends on such deep evidence integration. However, existing benchmarks fail to evaluate this capability in realistic workflows requiring multi-step evidence integration and expert-level judgment. To address this gap, we introduce MedProbeBench, the first benchmark leveraging high-quality clinical guidelines as expert-level references. Medical guidelines, with their rigorous standards in neutrality and verifiability, represent the pinnacle of medical expertise and pose substantial challenges for deep research agents. For evaluation, we propose MedProbe-Eval, a comprehensive evaluation framework featuring: (1) Holistic Rubrics with 1,200+ task-adaptive rubric criteria for comprehensive quality assessment, and (2) Fine-grained Evidence Verification for rigorous validation of evidence precision, grounded in 5,130+ atomic claims. Evaluation of 17 LLMs and deep research agents reveals critical gaps in evidence integration and guideline generation, underscoring the substantial distance between current capabilities and expert-level clinical guideline development. Project: https://github.com/uni-medical/MedProbeBench