# Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models

arXiv: https://arxiv.org/abs/2510.10278 · HTML: https://mlsanity.com/paper/2510.10278/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 11 Oct 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 17 of 17)

- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Identifying the Risks of LM Agents with an LM-Emulated Sandbox](https://mlsanity.com/paper/2309.15817/) (2023)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments](https://mlsanity.com/paper/2405.07960/) (2024)
- [MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs](https://mlsanity.com/paper/2501.17399/) (2025)
- [Time Travel in LLMs: Tracing Data Contamination in Large Language Models](https://mlsanity.com/paper/2308.08493/) (2023)
- [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://mlsanity.com/paper/2402.05201/) (2024)
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2401.16745/) (2024)
- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [A Survey on Multi-Turn Interaction Capabilities of Large Language Models](https://mlsanity.com/paper/2501.09959/) (2025)
- [A Foundational Framework and Methodology for Personalized Early and Timely Diagnosis](https://mlsanity.com/paper/2311.16195/) (2023)

## Built on by (most cited first, 9 of 9)

- [Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verification](https://mlsanity.com/paper/2603.02798/) (2026)
- [RealICU: Do LLM Agents Understand Long-Context ICU Data? A Benchmark Beyond Behavior Imitation](https://mlsanity.com/paper/2605.13542/) (2026)
- [DDX-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs](https://mlsanity.com/paper/2605.23629/) (2026)
- [Ask Before You Diagnose: Safe-Psych, a Sequential Evaluation Benchmark for LLMs in Psychiatry](https://mlsanity.com/paper/2607.13036/) (2026)
- [ClinDEF: A Dynamic Evaluation Framework for Large Language Models in Clinical Reasoning](https://mlsanity.com/paper/2512.23440/) (2025)
- [MedBench v5: A Dynamic, Process-Oriented, and Hallucination-Aware Benchmark for Clinical Multimodal Models](https://mlsanity.com/paper/2606.24155/) (2026)
- [CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning](https://mlsanity.com/paper/2606.31608/) (2026)
- [From Given to Gathered Evidence: Agentic Learning for Longitudinal Medical Reasoning](https://mlsanity.com/paper/2609.39566/) (2026)
- [Reinforcement Learning In Partially Verifiable Environments for Colon Cancer Treatment Planning](https://mlsanity.com/paper/2610.17426/) (2026)

## Abstract (arXiv)

Clinical reasoning in medicine is a hypothesis-driven process where physicians refine diagnoses from limited information through targeted history, physical examination, and diagnostic investigations. In contrast, current medical benchmarks for large language models (LLMs) primarily assess knowledge recall through single-turn questions, where complete clinical information is provided upfront. To address this gap, we introduce VivaBench, a multi-turn benchmark that evaluates sequential clinical reasoning in LLM agents. Our dataset consists of 1762 physician-curated clinical vignettes structured as interactive scenarios that simulate a (oral) examination in medical training, requiring agents to actively probe for relevant findings, select appropriate investigations, and synthesize information across multiple steps to reach a diagnosis. While current LLMs demonstrate competence in diagnosing conditions from well-described clinical presentations, their performance degrades significantly when required to navigate iterative diagnostic reasoning under uncertainty in our evaluation. Our analysis identified several failure modes that mirror common cognitive errors in clinical practice, including: (1) fixation on initial hypotheses, (2) inappropriate investigation ordering, (3) premature diagnostic closure, and (4) failing to screen for critical conditions. These patterns reveal fundamental limitations in how current LLMs reason and make decisions under uncertainty. Through VivaBench, we provide a standardized benchmark for evaluating conversational medical AI systems for real-world clinical decision support. Beyond medical applications, we contribute to the larger corpus of research on agentic AI by demonstrating how sequential reasoning trajectories can diverge in complex decision-making environments.