# Advancing Conversational Diagnostic AI with Multimodal Reasoning

arXiv: https://arxiv.org/abs/2505.04653 · HTML: https://mlsanity.com/paper/2505.04653/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 22 of its 71 references that are in the map.
- First posted on arXiv 6 May 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 22 of 22)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [Capabilities of Gemini Models in Medicine](https://mlsanity.com/paper/2404.18416/) (2024)
- [AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments](https://mlsanity.com/paper/2405.07960/) (2024)
- [GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI](https://mlsanity.com/paper/2408.03361/) (2024)
- [Towards Conversational Diagnostic AI](https://mlsanity.com/paper/2401.05654/) (2024)
- [ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogram](https://mlsanity.com/paper/2306.15681/) (2023)
- [Polaris: A Safety-focused LLM Constellation Architecture for Healthcare](https://mlsanity.com/paper/2403.13313/) (2024)
- [Enhancing Diagnostic Accuracy through Multi-Agent Conversations: Using Large Language Models to Mitigate Cognitive Bias](https://mlsanity.com/paper/2401.14589/) (2024)
- [Reasoning Like a Doctor: Improving Medical Dialogue Systems via Diagnostic Reasoning Process Alignment](https://mlsanity.com/paper/2406.13934/) (2024)
- [Active Acquisition for Multimodal Temporal Data: A Challenging Decision-Making Task](https://mlsanity.com/paper/2211.05039/) (2022)
- [Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications](https://mlsanity.com/paper/2411.03782/) (2024)
- [Conversational Medical AI: Ready for Practice](https://mlsanity.com/paper/2411.12808/) (2024)
- [Dialogue is Better Than Monologue: Instructing Medical LLMs via Strategical Conversations](https://mlsanity.com/paper/2501.17860/) (2025)
- [CheXphoto: 10,000+ Photos and Transformations of Chest X-rays for Benchmarking Deep Learning Robustness](https://mlsanity.com/paper/2007.06199/) (2020)
- [Automated Medical Report Generation for ECG Data: Bridging Medical Text and Signal Processing with Deep Learning](https://mlsanity.com/paper/2412.04067/) (2024)
- [Medical Dialogue Generation via Intuitive-then-Analytical Differential Diagnosis](https://mlsanity.com/paper/2401.06541/) (2024)
- [MINT: A wrapper to make multi-modal and multi-image AI models interactive](https://mlsanity.com/paper/2401.12032/) (2024)
- [Multimodal Marvels of Deep Learning in Medical Diagnosis: A Comprehensive Review of COVID-19 Detection](https://mlsanity.com/paper/2501.09506/) (2025)

## Built on by (most cited first, 20 of 20)

- [SymptomAI: Toward a Conversational AI Agent for Everyday Symptom Assessment](https://mlsanity.com/paper/2605.04012/) (2026)
- [Towards Conversational Medical AI with Eyes, Ears and a Voice](https://mlsanity.com/paper/2605.09272/) (2026)
- [MedVAL: Toward Expert-Level Medical Text Validation with Language Models](https://mlsanity.com/paper/2507.03152/) (2025)
- [A Survey of the State-of-the-Art in Conversational Question Answering Systems](https://mlsanity.com/paper/2509.05716/) (2025)
- [Format Inertia: A Failure Mechanism of LLMs in Medical Pre-Consultation](https://mlsanity.com/paper/2510.01688/) (2025)
- [Integrating Genomics into Multimodal EHR Foundation Models](https://mlsanity.com/paper/2510.23639/) (2025)
- [Note2Chat: Improving LLMs for Multi-Turn Clinical History Taking Using Medical Notes](https://mlsanity.com/paper/2601.21551/) (2026)
- [MedExAgent: Training LLM Agents to Ask, Examine, and Diagnose in Noisy Clinical Environments](https://mlsanity.com/paper/2605.07058/) (2026)
- [Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguese](https://mlsanity.com/paper/2606.07853/) (2026)
- [A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks](https://mlsanity.com/paper/2607.02175/) (2026)
- [Fairness Evaluation of Large Language Models in Academic Library Reference Services](https://mlsanity.com/paper/2507.04224/) (2025)
- [Towards Better Health Conversations: The Benefits of Context-seeking](https://mlsanity.com/paper/2510.18880/) (2025)
- [Dialogue to Question Generation for Evidence-based Medical Guideline Agent Development](https://mlsanity.com/paper/2603.23937/) (2026)
- [The complexities of patient-centred conversational artificial intelligence](https://mlsanity.com/paper/2607.08625/) (2026)
- [PATHFinder Agent for Tailored Prenatal Care](https://mlsanity.com/paper/2607.24768/) (2026)
- [Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases](https://mlsanity.com/paper/2607.25933/) (2026)
- [CoM3eT: A foundation model for medical image analysis through federated, multidimensional context integration](https://mlsanity.com/paper/2608.16268/) (2026)
- [A Source-Grounded Framework for Constructing and Evaluating Progressive Multimodal Diagnostic Dialogues from Clinical Case Reports](https://mlsanity.com/paper/2608.22713/) (2026)
- [DischargeBench: Pre-Deployment Evaluation of LLM Discharge Educators through Persona-Grounded Patient Simulation](https://mlsanity.com/paper/2609.20827/) (2026)
- [DynamicDx: Evaluating Evidence Acquisition in Video-Based Diagnosis](https://mlsanity.com/paper/2609.32957/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated great potential for conducting diagnostic conversations but evaluation has been largely limited to language-only interactions, deviating from the real-world requirements of remote care delivery. Instant messaging platforms permit clinicians and patients to upload and discuss multimodal medical artifacts seamlessly in medical consultation, but the ability of LLMs to reason over such data while preserving other attributes of competent diagnostic conversation remains unknown. Here we advance the conversational diagnosis and management performance of the Articulate Medical Intelligence Explorer (AMIE) through a new capability to gather and interpret multimodal data, and reason about this precisely during consultations. Leveraging Gemini 2.0 Flash, our system implements a state-aware dialogue framework, where conversation flow is dynamically controlled by intermediate model outputs reflecting patient states and evolving diagnoses. Follow-up questions are strategically directed by uncertainty in such patient states, leading to a more structured multimodal history-taking process that emulates experienced clinicians. We compared AMIE to primary care physicians (PCPs) in a randomized, blinded, OSCE-style study of chat-based consultations with patient actors. We constructed 105 evaluation scenarios using artifacts like smartphone skin photos, ECGs, and PDFs of clinical documents across diverse conditions and demographics. Our rubric assessed multimodal capabilities and other clinically meaningful axes like history-taking, diagnostic accuracy, management reasoning, communication, and empathy. Specialist evaluation showed AMIE to be superior to PCPs on 7/9 multimodal and 29/32 non-multimodal axes (including diagnostic accuracy). The results show clear progress in multimodal conversational diagnostic AI, but real-world translation needs further research.