mlsanityData to 7 Oct 2026

Towards Conversational Diagnostic AI

arXiv 2401.05654 ↗First posted 11 Jan 2024cs.AI cs.CL cs.LGcited by 79

Not on the map. Why?

Abstract (arXiv)

At the heart of medicine lies the physician-patient dialogue, where skillful history-taking paves the way for accurate diagnosis, effective management, and enduring trust. Artificial Intelligence (AI) systems capable of diagnostic dialogue could increase accessibility, consistency, and quality of care. However, approximating clinicians' expertise is an outstanding grand challenge. Here, we introduce AMIE (Articulate Medical Intelligence Explorer), a Large Language Model (LLM) based AI system optimized for diagnostic dialogue. AMIE uses a novel self-play based simulated environment with automated feedback mechanisms for scaling learning across diverse disease conditions, specialties, and contexts. We designed a framework for evaluating clinically-meaningful axes of performance including history-taking, diagnostic accuracy, management reasoning, communication skills, and empathy. We compared AMIE's performance to that of primary care physicians (PCPs) in a randomized, double-blind crossover study of text-based consultations with validated patient actors in the style of an Objective Structured Clinical Examination (OSCE). The study included 149 case scenarios from clinical providers in Canada, the UK, and India, 20 PCPs for comparison with AMIE, and evaluations by specialist physicians and patient actors. AMIE demonstrated greater diagnostic accuracy and superior performance on 28 of 32 axes according to specialist physicians and 24 of 26 axes according to patient actors. Our research has several limitations and should be interpreted with appropriate caution. Clinicians were limited to unfamiliar synchronous text-chat which permits large-scale LLM-patient interactions but is not representative of usual clinical practice. While further research is required before AMIE could be translated to real-world settings, the results represent a milestone towards conversational diagnostic AI.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Builds on 44 of its 121 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

44 of 121 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  5. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2023PaLM 2 Technical Reportcited by 1,154
  8. 2018Model Cards for Model Reportingcited by 1,068
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2022Red Teaming Language Models with Language Modelscited by 983
  11. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  12. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
32 more
  1. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  2. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  4. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  5. 2022Self-critiquing models for assisting human evaluatorscited by 372
  6. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  7. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  8. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  9. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  10. 2020A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Supportcited by 182
  11. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 169
  12. 2023Conformal Language Modelingcited by 163
  13. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  14. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 157
  15. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  16. 2023Training Language Models with Language Feedback at Scalecited by 125
  17. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
  18. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  19. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 115
  20. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
  21. 2022Interactive Model Cards: A Human-Centered Approach to Model Documentationcited by 61
  22. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  23. 2021Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?cited by 46
  24. 2023Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Promptscited by 45
  25. 2023Decision-Oriented Dialogue for Human-AI Collaborationcited by 29
  26. 2021ReMeDi: Resources for Multi-domain, Multi-service, Medical Dialoguescited by 20
  27. 2017Speech recognition for medical conversationscited by 18
  28. 2023Fairness in Language Models Beyond English: Gaps and Challengescited by 16
  29. 2023Uncertainty-aware Language Modeling for Selective Question Answeringcited by 16
  30. 2022DialMed: A Dataset for Dialogue-based Medication Recommendationcited by 12
  31. 2023Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settingscited by 8
  32. 2022CDialog: A Multi-turn Covid-19 Conversation Dataset for Entity-Aware Dialog Generationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 37; 2025: 27; 2026: 15.

79 papers.

Sort:

By area:

  1. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  2. 2024Capabilities of Gemini Models in Medicinecited by 231
  3. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  4. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  5. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 78
  6. 2024Building Machines that Learn and Think with Peoplecited by 73
  7. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  8. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 64
  9. 2024Autonomous LLM-driven research from data to human-verifiable research paperscited by 50
  10. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  11. 2024MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Modelscited by 43
  12. 2024Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulatorcited by 36
38 more
  1. 2024Large Language Models Produce Responses Perceived to be Empathiccited by 36
  2. 2024Zero-Shot Clinical Trial Patient Matching with LLMscited by 35
  3. 2024Small Language Models Learn Enhanced Reasoning Skills from Medical Textbookscited by 30
  4. 2025A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domainscited by 28
  5. 2024Polaris: A Safety-focused LLM Constellation Architecture for Healthcarecited by 27
  6. 2024Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementationcited by 27
  7. 2024Do LLMs estimate uncertainty well in instruction-following?cited by 26
  8. 2024Do LLMs "know" internally when they follow instructions?cited by 25
  9. 2025RadVLM: A Multitask Conversational Vision-Language Model for Radiologycited by 24
  10. 2024Mitigating the Risk of Health Inequity Exacerbated by Large Language Modelscited by 21
  11. 2024Leveraging Large Language Model as Simulated Patients for Clinical Educationcited by 20
  12. 2024ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agentscited by 20
  13. 2025Advancing Conversational Diagnostic AI with Multimodal Reasoningcited by 20
  14. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  15. 2024Memorization in deep learning: A surveycited by 14
  16. 2025A Scalable Framework for Evaluating Health Language Modelscited by 11
  17. 2024Development of a Large Language Model-based Multi-Agent Clinical Decision Support System for Korean Triage and Acuity Scale (KTAS)-Based Triage and Treatment Planning in Emergency Departmentscited by 10
  18. 2025End-to-End Agentic RAG System Training for Traceable Diagnostic Reasoningcited by 10
  19. 2025The Illusion of Readiness in Health AIcited by 10
  20. 2026Baichuan-M3: Modeling Clinical Inquiry for Reliable Medical Decision-Makingcited by 10
  21. 2025MedMMV: A Controllable Multimodal Multi-Agent Framework for Reliable and Verifiable Clinical Reasoningcited by 8
  22. 2025A Survey of Inductive Reasoning for Large Language Modelscited by 8
  23. 2025Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare Safetycited by 7
  24. 2025Benchmarking Vision, Language, & Action Models in Procedurally Generated, Open Ended Action Environmentscited by 6
  25. 2024PsyDI: Towards a Personalized and Progressively In-depth Chatbot for Psychological Measurementscited by 5
  26. 2025Position: Stop Acting Like Language Model Agents Are Normal Agentscited by 5
  27. 2025AI Agents for Conversational Patient Triage: Preliminary Simulation-Based Evaluation with Real-World EHR Datacited by 5
  28. 2025KnowGuard: Knowledge-Driven Abstention for Multi-Round Clinical Reasoningcited by 5
  29. 2024Attribute Structuring Improves LLM-Based Evaluation of Clinical Text Summariescited by 4
  30. 2024Interpretable Differential Diagnosis with Dual-Inference Large Language Modelscited by 4
  31. 2024Jailbreaking LLM-Controlled Robotscited by 4
  32. 2024Beyond Multiple-Choice Accuracy: Real-World Challenges of Implementing Large Language Models in Healthcarecited by 4
  33. 2024The Minimum Information about CLinical Artificial Intelligence Checklist for Generative Modeling Research (MI-CLAIM-GEN)cited by 3
  34. 2024Conversational Disease Diagnosis via External Planner-Controlled Large Language Modelscited by 3
  35. 2024Language Models Meet Anomaly Detection for Better Interpretability and Generalizabilitycited by 3
  36. 2024RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignmentcited by 3
  37. 2025Improving Interactive Diagnostic Ability of a Large Language Model Agent Through Clinical Experience Learningcited by 3
  38. 2024Selective Fine-tuning on LLM-labeled Data May Reduce Reliance on Human Annotation: A Case Study Using Schedule-of-Event Table Detectioncited by 2

50 most cited of 79 All 79 (JSON)