# MedConsultBench: A Full-Cycle, Fine-Grained, Process-Aware Benchmark for Medical Consultation Agents

arXiv: https://arxiv.org/abs/2601.12661 · HTML: https://mlsanity.com/paper/2601.12661/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 103 references that are in the map.
- First posted on arXiv 19 Jan 2026.

## Builds on (most cited first, 17 of 17)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments](https://mlsanity.com/paper/2405.07960/) (2024)
- [MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding](https://mlsanity.com/paper/2501.18362/) (2025)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs](https://mlsanity.com/paper/2501.17399/) (2025)
- [Large Language Models are Few-Shot Health Learners](https://mlsanity.com/paper/2305.15525/) (2023)
- [Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator](https://mlsanity.com/paper/2403.08495/) (2024)
- [Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery](https://mlsanity.com/paper/2304.13714/) (2023)
- [Scaling-up Empirical Risk Minimization: Optimization of Incomplete U-statistics](https://mlsanity.com/paper/1501.02629/) (2015)
- [Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm](https://mlsanity.com/paper/2403.16446/) (2024)
- [The Dialogue That Heals: A Comprehensive Evaluation of Doctor Agents' Inquiry Capability](https://mlsanity.com/paper/2509.24958/) (2025)

## Built on by (most cited first, 4 of 4)

- [MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation](https://mlsanity.com/paper/2607.09142/) (2026)
- [MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agents](https://mlsanity.com/paper/2607.18999/) (2026)
- [ConsultMind:Towards Automated Diagnostic Consultation via Uncertainty-Aware Reasoning](https://mlsanity.com/paper/2609.30796/) (2026)
- [KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy](https://mlsanity.com/paper/2609.38480/) (2026)

## Abstract (arXiv)

Current evaluations of medical consultation agents often prioritize outcome-oriented tasks, frequently overlooking the end-to-end process integrity and clinical safety essential for real-world practice. While recent interactive benchmarks have introduced dynamic scenarios, they often remain fragmented and coarse-grained, failing to capture the structured inquiry logic and diagnostic rigor required in professional consultations. To bridge this gap, we propose MedConsultBench, a comprehensive framework designed to evaluate the complete online consultation cycle by covering the entire clinical workflow from history taking and diagnosis to treatment planning and follow-up Q&A. Our methodology introduces Atomic Information Units (AIUs) to track clinical information acquisition at a sub-turn level, enabling precise monitoring of how key facts are elicited through 22 fine-grained metrics. By addressing the underspecification and ambiguity inherent in online consultations, the benchmark evaluates uncertainty-aware yet concise inquiry while emphasizing medication regimen compatibility and the ability to handle realistic post-prescription follow-up Q&A via constraint-respecting plan revisions. Systematic evaluation of 19 large language models reveals that high diagnostic accuracy often masks significant deficiencies in information-gathering efficiency and medication safety. These results underscore a critical gap between theoretical medical knowledge and clinical practice ability, establishing MedConsultBench as a rigorous foundation for aligning medical AI with the nuanced requirements of real-world clinical care.