MedDialBench: Benchmarking LLM Diagnostic Robustness under Parametric Adversarial Patient Behaviors
Not on the map. Why?
Abstract (arXiv)
Interactive medical dialogue benchmarks have shown that LLM diagnostic accuracy degrades significantly when interacting with non-cooperative patients, yet existing approaches either apply adversarial behaviors without graded severity or case-specific grounding, or reduce patient non-cooperation to a single ungraded axis, and none analyze cross-dimension interactions. We introduce MedDialBench, a benchmark enabling controlled, dose-response characterization of how individual patient behavior dimensions affect LLM diagnostic robustness. It decomposes patient behavior into five dimensions -- Logic Consistency, Health Cognition, Expression Style, Disclosure, and Attitude -- each with graded severity levels and case-specific behavioral scripts. This controlled factorial design enables graded sensitivity analysis, dose-response profiling, and cross-dimension interaction detection. Evaluating five frontier LLMs across 7,225 dialogues (85 cases x 17 configurations x 5 models), we find a fundamental asymmetry: information pollution (fabricating symptoms) produces 1.7-3.4x larger accuracy drops than information deficit (withholding information), and fabricating is the only configuration achieving statistical significance across all five models (McNemar p < 0.05). Among six dimension combinations, fabricating is the sole driver of super-additive interaction: all three fabricating-involving pairs produce O/E ratios of 0.70-0.81 (35-44% of eligible cases fail under the combination despite succeeding under each dimension alone), while all non-fabricating pairs show purely additive effects (O/E ~ 1.0). Inquiry strategy moderates deficit but not pollution: exhaustive questioning recovers withheld information, but cannot compensate for fabricated inputs. Models exhibit distinct vulnerability profiles, with worst-case drops ranging from 38.8 to 54.1 percentage points.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 16 of its 26 references that are in the map. First posted on arXiv 8 Apr 2026.
Builds on
16 of 26 references.
References · 16
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
- 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
- 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
- 2024Capabilities of Gemini Models in Medicinecited by 237
- 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 231
- 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 154
- 2024MedFuzz: Exploring the Robustness of Large Language Models in Medical Question Answeringcited by 26
- 2024Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical educationcited by 13
- 2026MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Modelscited by 12
- 2025The Dialogue That Heals: A Comprehensive Evaluation of Doctor Agents' Inquiry Capabilitycited by 10
- 2025Clinical knowledge in LLMs does not translate to human interactionscited by 7
4 more
- 2026SycoEval-EM: Sycophancy Evaluation of Large Language Models in Simulated Clinical Encounters for Emergency Carecited by 5
- 2026LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosiscited by 4
- 2026Beyond Idealized Patients: Evaluating LLMs under Challenging Patient Behaviors in Medical Consultationscited by 3
- 2025MedPI: Evaluating AI Systems in Medical Patient-facing Interactionscited by 2
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026ClinEnv: An Interactive Multi-Stage Long Horizon EHR Environment for Agentscited by 4
- 2026MedBench v5: A Dynamic, Process-Oriented, and Hallucination-Aware Benchmark for Clinical Multimodal Modelscited by 0
- 2026MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultationcited by 0
- 2026Measuring Stability and Failure Behavior in Language Models Under Structured Perturbationscited by 0
- 2026Measuring Stability and Failure Behavior in Language Models Under Structured Perturbations22 Aug 2026
- 2026MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation10 Jul 2026
- 2026MedBench v5: A Dynamic, Process-Oriented, and Hallucination-Aware Benchmark for Clinical Multimodal Models23 Jun 2026
- 2026ClinEnv: An Interactive Multi-Stage Long Horizon EHR Environment for Agents1 Jun 2026