# MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Models

arXiv: https://arxiv.org/abs/2601.03023 · HTML: https://mlsanity.com/paper/2601.03023/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Builds on (most cited first, 18 of 18)

- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments](https://mlsanity.com/paper/2405.07960/) (2024)
- [Qwen2.5-1M Technical Report](https://mlsanity.com/paper/2501.15383/) (2025)
- [MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning](https://mlsanity.com/paper/2406.00922/) (2024)
- [AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator](https://mlsanity.com/paper/2402.09742/) (2024)
- [MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks](https://mlsanity.com/paper/2505.23802/) (2025)
- [Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactions](https://mlsanity.com/paper/2503.22678/) (2025)
- [Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator](https://mlsanity.com/paper/2403.08495/) (2024)
- [LLM-MedQA: Enhancing Medical Question Answering through Case Studies in Large Language Models](https://mlsanity.com/paper/2501.05464/) (2024)
- [PATIENT-Psi: Using Large Language Models to Simulate Patients for Training Mental Health Professionals](https://mlsanity.com/paper/2405.19660/) (2024)
- [3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark](https://mlsanity.com/paper/2504.13861/) (2025)
- [LLM-Mini-CEX: Automatic Evaluation of Large Language Model for Diagnostic Conversation](https://mlsanity.com/paper/2308.07635/) (2023)
- [An Automatic Evaluation Framework for Multi-turn Medical Consultations Capabilities of Large Language Models](https://mlsanity.com/paper/2309.02077/) (2023)
- [The Dialogue That Heals: A Comprehensive Evaluation of Doctor Agents' Inquiry Capability](https://mlsanity.com/paper/2509.24958/) (2025)
- [Dynamic Affective Memory Management for Personalized LLM Agents](https://mlsanity.com/paper/2510.27418/) (2025)

## Built on by (most cited first, 12 of 12)

- [HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats](https://mlsanity.com/paper/2604.27470/) (2026)
- [Interactive Benchmarks](https://mlsanity.com/paper/2603.04737/) (2026)
- [LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosis](https://mlsanity.com/paper/2602.09379/) (2026)
- [MedDialBench: Benchmarking LLM Diagnostic Robustness under Parametric Adversarial Patient Behaviors](https://mlsanity.com/paper/2604.06846/) (2026)
- [Information-seeking failures of large language models in agentic clinical reasoning](https://mlsanity.com/paper/2607.10275/) (2026)
- [MedConceal: A Benchmark for Clinical Hidden-Concern Reasoning Under Partial Observability](https://mlsanity.com/paper/2604.08788/) (2026)
- [The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy](https://mlsanity.com/paper/2607.11175/) (2026)
- [MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters](https://mlsanity.com/paper/2607.08257/) (2026)
- [Toward Workflow-Aware Benchmarking for Healthcare NLP Agents](https://mlsanity.com/paper/2609.00296/) (2026)
- [MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation](https://mlsanity.com/paper/2607.09142/) (2026)
- [PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents](https://mlsanity.com/paper/2607.25485/) (2026)
- [Quantifying the Relationship Between Clinical Safety and Environmental Impact in Therapeutic LLMs](https://mlsanity.com/paper/2608.11830/) (2026)

## Abstract (arXiv)

Medical conversational AI (AI) plays a pivotal role in the development of safer and more effective medical dialogue systems. However, existing benchmarks and evaluation frameworks for assessing the information-gathering and diagnostic reasoning abilities of medical large language models (LLMs) have not been rigorously evaluated. To address these gaps, we present MedDialogRubrics, a novel benchmark comprising 5,200 synthetically constructed patient cases and over 60,000 fine-grained evaluation rubrics generated by LLMs and subsequently refined by clinical experts, specifically designed to assess the multi-turn diagnostic capabilities of LLM. Our framework employs a multi-agent system to synthesize realistic patient records and chief complaints from underlying disease knowledge without accessing real-world electronic health records, thereby mitigating privacy and data-governance concerns. We design a robust Patient Agent that is limited to a set of atomic medical facts and augmented with a dynamic guidance mechanism that continuously detects and corrects hallucinations throughout the dialogue, ensuring internal coherence and clinical plausibility of the simulated cases. Furthermore, we propose a structured LLM-based and expert-annotated rubric-generation pipeline that retrieves Evidence-Based Medicine (EBM) guidelines and utilizes the reject sampling to derive a prioritized set of rubric items ("must-ask" items) for each case. We perform a comprehensive evaluation of state-of-the-art models and demonstrate that, across multiple assessment dimensions, current models face substantial challenges. Our results indicate that improving medical dialogue will require advances in dialogue management architectures, not just incremental tuning of the base-model.