Rethinking Patient Education as Multi-turn Multi-modal Interaction
Not on the map. Why?
Abstract (arXiv)
Most medical multimodal benchmarks focus on static tasks such as image question answering, report generation, and plain-language rewriting. Patient education is more demanding: systems must identify relevant evidence across images, show patients where to look, explain findings in accessible language, and handle confusion or distress. Yet most patient education work remains text-only, even though combined image-and-text explanations may better support understanding. We introduce MedImageEdu, a benchmark for multi-turn, evidence-grounded radiology patient education. Each case provides a radiology report with report text and case images. A DoctorAgent interacts with a PatientAgent, conditioned on a hidden profile that captures factors such as education level, health literacy, and personality. When a patient question would benefit from visual support, the DoctorAgent can issue drawing instructions grounded in the report, case images, and the current question to a benchmark-provided drawing tool. The tool returns image(s), after which the DoctorAgent produces a final multimodal response consisting of the image(s) and a grounded plain-language explanation. MedImageEdu contains 150 cases from three sources and evaluates both the consultation process and the final multimodal response along five dimensions: Consultation, Safety and Scope, Language Quality, Drawing Quality, and Image-Text Response Quality. Across representative open- and closed-source vision-language model agents, we find three consistent gaps: fluent language often outpaces faithful visual grounding, safety is the weakest dimension across disease categories, and emotionally tense interactions are harder than low education or low health literacy. MedImageEdu provides a controlled testbed for assessing whether multimodal agents can teach from evidence rather than merely answer from text.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 66 references that are in the map. First posted on arXiv 16 Apr 2026.
Builds on
38 of 66 references.
References · 38
- 2023GPT-4 Technical Reportcited by 18,132
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
- 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
- 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
- 2025MedGemma Technical Reportcited by 398
- 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
- 2023GPTScore: Evaluate as You Desirecited by 357
- 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
- 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
26 more
- 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
- 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
- 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
- 2025MedRAX: Medical Reasoning Agent for Chest X-raycited by 68
- 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 65
- 2023NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notescited by 48
- 2023A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluatorscited by 45
- 2026MedGemma 1.5 Technical Reportcited by 43
- 2023Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potentialcited by 39
- 2024Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Modelscited by 34
- 2024RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Modelscited by 24
- 2023PaniniQA: Enhancing Patient Education Through Interactive Question Answeringcited by 21
- 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Modelscited by 21
- 2022MedJEx: A Medical Jargon Extraction Model with Wiki's Hyperlink Span and Contextualized Masked Language Model Scorecited by 20
- 2024Comparing Two Model Designs for Clinical Note Generation; Is an LLM a Useful Evaluator of Consistency?cited by 20
- 2023README: Bridging Medical Jargon and Lay Understanding for Patient Education through Data-Centric NLPcited by 17
- 2024MedQA-CS: Objective Structured Clinical Examination (OSCE)-Style Benchmark for Evaluating LLM Clinical Skillscited by 16
- 2024MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedbackcited by 14
- 2025S-Chain: Structured Visual Chain-of-Thought For Medicinecited by 10
- 2025DischargeSim: A Simulation Benchmark for Educational Doctor-Patient Communication at Dischargecited by 7
- 2025MedReadCtrl: Personalizing medical text generation with readability-controlled instruction learningcited by 5
- 2024MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosiscited by 4
- 2025Chatbot To Help Patients Understand Their Healthcited by 4
- 2025ChatThero: An LLM-Supported Chatbot for Behavior Change and Therapeutic Support in Addiction Recoverycited by 3
- 2026Medical thinking with multiple imagescited by 3
- 2025ChatCLIDS: Simulating Persuasive AI Dialogues to Promote Closed-Loop Insulin Adoption in Type 1 Diabetes Carecited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026MedQA-MM: Shortcuts Behind Medical Visual Reasoning
- 2025MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text
- 2025Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations
- 2023RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance
- 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
- 2026IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations
- 2026Concept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generation
- 2025MedGEN-Bench: A Contextually Entangled Benchmark for Open-ended Multimodal Medical Generation