# Shallow Robustness, Deep Vulnerabilities: Multi-Turn Evaluation of Medical LLMs

arXiv: https://arxiv.org/abs/2510.12255 · HTML: https://mlsanity.com/paper/2510.12255/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 15 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Builds on (most cited first, 11 of 11)

- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [MedGemma Technical Report](https://mlsanity.com/paper/2507.05201/) (2025)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments](https://mlsanity.com/paper/2405.07960/) (2024)
- [Large language models in healthcare and medical domain: A review](https://mlsanity.com/paper/2401.06775/) (2023)
- [MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering](https://mlsanity.com/paper/2404.05590/) (2024)
- [Addressing cognitive bias in medical language models](https://mlsanity.com/paper/2402.08113/) (2024)
- [How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark](https://mlsanity.com/paper/2505.18761/) (2025)
- [Exploring the Robustness of Large Language Models for Solving Programming Problems](https://mlsanity.com/paper/2306.14583/) (2023)
- [Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine](https://mlsanity.com/paper/2411.14487/) (2024)
- [Enhancing Clinical Multiple-Choice Questions Benchmarks with Knowledge Graph Guided Distractor Generation](https://mlsanity.com/paper/2506.00612/) (2025)

## Built on by (most cited first, 8 of 8)

- [Why LLMs Give In: Conversational Factors and Reasoning Behind Medical Sycophancy](https://mlsanity.com/paper/2608.01017/) (2026)
- [ThReadMed-QA: A Multi-Turn Medical Dialogue Benchmark from Real Patient Questions](https://mlsanity.com/paper/2603.11281/) (2026)
- [QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating Large Language Models](https://mlsanity.com/paper/2603.13691/) (2026)
- [Evaluating Large Language Models on Misconceptions in Multi-Turn Medical Conversations](https://mlsanity.com/paper/2607.12884/) (2026)
- [Untangling the Mechanisms of Misleading Context in Medical Question Answering](https://mlsanity.com/paper/2609.02754/) (2026)
- [Finishing the Task Is Not Enough: Evaluating Agent Resilience and Considerate Participation under Accumulating Challenge](https://mlsanity.com/paper/2609.10724/) (2026)
- [MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations](https://mlsanity.com/paper/2609.12851/) (2026)
- [Unanimously Wrong: Certified Abstention from How Medical LLM Consensus Forms](https://mlsanity.com/paper/2610.07570/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are rapidly transitioning into medical clinical use, yet their reliability under realistic, multi-turn interactions remains poorly understood. Existing evaluation frameworks typically assess single-turn question answering under idealized conditions, overlooking the complexities of medical consultations where conflicting input, misleading context, and authority influence are common. We introduce MedQA-Followup, a framework for systematically evaluating multi-turn robustness in medical question answering. Our approach distinguishes between shallow robustness (resisting misleading initial context) and deep robustness (maintaining accuracy when answers are challenged across turns), while also introducing an indirect-direct axis that separates contextual framing (indirect) from explicit suggestion (direct). Using controlled interventions on the MedQA dataset, we evaluate five state-of-the-art LLMs and find that while models perform reasonably well under shallow perturbations, they exhibit severe vulnerabilities in multi-turn settings, with accuracy dropping from 91.2% to as low as 13.5% for Claude Sonnet 4. Counterintuitively, indirect, context-based interventions are often more harmful than direct suggestions, yielding larger accuracy drops across models and exposing a significant vulnerability for clinical deployment. Further compounding analyses reveal model differences, with some showing additional performance drops under repeated interventions while others partially recovering or even improving. These findings highlight multi-turn robustness as a critical but underexplored dimension for safe and reliable deployment of medical LLMs.