# Conversational Medical AI: Ready for Practice

arXiv: https://arxiv.org/abs/2411.12808 · HTML: https://mlsanity.com/paper/2411.12808/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 33 references that are in the map.
- First posted on arXiv 19 Nov 2024.

## Builds on (most cited first, 7 of 7)

- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [Small LLMs Are Weak Tool Learners: A Multi-LLM Agent](https://mlsanity.com/paper/2401.07324/) (2024)
- [The Ethics of ChatGPT in Medicine and Healthcare: A Systematic Review on Large Language Models (LLMs)](https://mlsanity.com/paper/2403.14473/) (2024)
- [DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains](https://mlsanity.com/paper/2304.00958/) (2023)
- [Adopting AI: How Familiarity Breeds Both Trust and Contempt](https://mlsanity.com/paper/2305.01405/) (2023)

## Built on by (most cited first, 4 of 4)

- [Advancing Conversational Diagnostic AI with Multimodal Reasoning](https://mlsanity.com/paper/2505.04653/) (2025)
- [AI Agents for Conversational Patient Triage: Preliminary Simulation-Based Evaluation with Real-World EHR Data](https://mlsanity.com/paper/2506.04032/) (2025)
- [Patient-Zero: Scaling Synthetic Patient Agents to Real-World Distributions without Real Patient Data](https://mlsanity.com/paper/2509.11078/) (2025)
- [How people use Copilot for Health](https://mlsanity.com/paper/2604.15331/) (2026)

## Abstract (arXiv)

The shortage of doctors is creating a critical squeeze in access to medical expertise. While conversational Artificial Intelligence (AI) holds promise in addressing this problem, its safe deployment in patient-facing roles remains largely unexplored in real-world medical settings. We present the first large-scale evaluation of a physician-supervised LLM-based conversational agent in a real-world medical setting. Our agent, Mo, was integrated into an existing medical advice chat service. Over a three-week period, we conducted a randomized controlled experiment with 926 cases to evaluate patient experience and satisfaction. Among these, Mo handled 298 complete patient interactions, for which we report physician-assessed measures of safety and medical accuracy. Patients reported higher clarity of information (3.73 vs 3.62 out of 4, p < 0.05) and overall satisfaction (4.58 vs 4.42 out of 5, p < 0.05) with AI-assisted conversations compared to standard care, while showing equivalent levels of trust and perceived empathy. The high opt-in rate (81% among respondents) exceeded previous benchmarks for AI acceptance in healthcare. Physician oversight ensured safety, with 95% of conversations rated as "good" or "excellent" by general practitioners experienced in operating a medical advice chat service. Our findings demonstrate that carefully implemented AI medical assistants can enhance patient experience while maintaining safety standards through physician supervision. This work provides empirical evidence for the feasibility of AI deployment in healthcare communication and insights into the requirements for successful integration into existing healthcare services.