# Towards Expert-level Medical AI for Real-time Video Consultations

arXiv: https://arxiv.org/abs/2608.09861 · HTML: https://mlsanity.com/paper/2608.09861/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 81 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Builds on (most cited first, 19 of 19)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [Capabilities of Gemini Models in Medicine](https://mlsanity.com/paper/2404.18416/) (2024)
- [Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming](https://mlsanity.com/paper/2408.16725/) (2024)
- [Agents Thinking Fast and Slow: A Talker-Reasoner Architecture](https://mlsanity.com/paper/2410.08328/) (2024)
- [Polaris: A Safety-focused LLM Constellation Architecture for Healthcare](https://mlsanity.com/paper/2403.13313/) (2024)
- [HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats](https://mlsanity.com/paper/2604.27470/) (2026)
- [AI-based Clinical Decision Support for Primary Care: A Real-World Study](https://mlsanity.com/paper/2507.16947/) (2025)
- [KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI](https://mlsanity.com/paper/2510.02327/) (2025)
- [A prospective clinical feasibility study of a conversational diagnostic AI in an ambulatory primary care clinic](https://mlsanity.com/paper/2603.08448/) (2026)
- [Towards physician-centered oversight of conversational diagnostic AI](https://mlsanity.com/paper/2507.15743/) (2025)
- [Towards Conversational Medical AI with Eyes, Ears and a Voice](https://mlsanity.com/paper/2605.09272/) (2026)
- [Towards Synthetic Data Generation for Improved Pain Recognition in Videos under Patient Constraints](https://mlsanity.com/paper/2409.16382/) (2024)
- [Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs](https://mlsanity.com/paper/2605.12460/) (2026)
- [ResidencyRL: Reinforcement Learning in Simulated Clinical Environments](https://mlsanity.com/paper/2608.07418/) (2026)
- [Real-Time Voice AI Hears but Does Not Listen](https://mlsanity.com/paper/2606.26083/) (2026)

## Abstract (arXiv)

Audio-visual interaction is the standard for patient-physician consultations, enabling natural communication and effective assessment of illness through non-verbal cues. While text-based AI has shown promise, it discards essential perceptual dimensions and limits patients who cannot articulate symptoms in writing. Early efforts to extend medical AI to audio-visual interaction have demonstrated feasibility but not reached clinician-level performance. Here, we provide the first demonstration of expert-level AI in real-time clinical video consultations using AMIE (Articulate Medical Intelligence Explorer) in a video configuration. AMIE (Video) is a Gemini-based multi-agent system integrating low-latency dialogue, clinical reasoning, and real-time audio-visual perception. To guide development, we established a taxonomy and automated evaluations for clinical audio-visual cues in telehealth settings. In a randomized Objective Structured Clinical Examination (OSCE) study with 30 primary care physicians (PCPs), 15 patient actors and 100 clinical scenarios, we compared AMIE (Video), its text-only counterpart AMIE (Text), and PCPs consulting via video. Clinical evaluators rated AMIE (Video) on par or better than PCPs in history-taking, diagnosis, management, and physical observation and examination. Patient actors preferred AMIE's approach to assessing and explaining conditions, while PCPs were preferred for rapport and partnership building. In modality ablation, patient actors preferred AMIE (Video)'s interface over text chat for communicative effectiveness, convenience, and feeling understood. Limitations remain in fine anatomical precision, subtle affective nuances, and high-frequency movements. While further research is needed before real-world translation, these results mark an important milestone toward AI systems capable of augmenting care across the sensory complexity of clinical practice.