mlsanityData to 7 Oct 2026

Rethinking Patient Education as Multi-turn Multi-modal Interaction

arXiv 2604.14656 ↗First posted 16 Apr 2026cs.AI cs.CL cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Most medical multimodal benchmarks focus on static tasks such as image question answering, report generation, and plain-language rewriting. Patient education is more demanding: systems must identify relevant evidence across images, show patients where to look, explain findings in accessible language, and handle confusion or distress. Yet most patient education work remains text-only, even though combined image-and-text explanations may better support understanding. We introduce MedImageEdu, a benchmark for multi-turn, evidence-grounded radiology patient education. Each case provides a radiology report with report text and case images. A DoctorAgent interacts with a PatientAgent, conditioned on a hidden profile that captures factors such as education level, health literacy, and personality. When a patient question would benefit from visual support, the DoctorAgent can issue drawing instructions grounded in the report, case images, and the current question to a benchmark-provided drawing tool. The tool returns image(s), after which the DoctorAgent produces a final multimodal response consisting of the image(s) and a grounded plain-language explanation. MedImageEdu contains 150 cases from three sources and evaluates both the consultation process and the final multimodal response along five dimensions: Consultation, Safety and Scope, Language Quality, Drawing Quality, and Image-Text Response Quality. Across representative open- and closed-source vision-language model agents, we find three consistent gaps: fluent language often outpaces faithful visual grounding, safety is the weakest dimension across disease categories, and emotionally tense interactions are harder than low education or low health literacy. MedImageEdu provides a controlled testbed for assessing whether multimodal agents can teach from evidence rather than merely answer from text.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 66 references that are in the map. First posted on arXiv 16 Apr 2026.

Builds on

38 of 66 references.

References · 38

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  6. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  7. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  8. 2025MedGemma Technical Reportcited by 398
  9. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  10. 2023GPTScore: Evaluate as You Desirecited by 357
  11. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  12. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
26 more
  1. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  2. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  3. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  4. 2025MedRAX: Medical Reasoning Agent for Chest X-raycited by 68
  5. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 65
  6. 2023NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notescited by 48
  7. 2023A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluatorscited by 45
  8. 2026MedGemma 1.5 Technical Reportcited by 43
  9. 2023Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potentialcited by 39
  10. 2024Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Modelscited by 34
  11. 2024RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Modelscited by 24
  12. 2023PaniniQA: Enhancing Patient Education Through Interactive Question Answeringcited by 21
  13. 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Modelscited by 21
  14. 2022MedJEx: A Medical Jargon Extraction Model with Wiki's Hyperlink Span and Contextualized Masked Language Model Scorecited by 20
  15. 2024Comparing Two Model Designs for Clinical Note Generation; Is an LLM a Useful Evaluator of Consistency?cited by 20
  16. 2023README: Bridging Medical Jargon and Lay Understanding for Patient Education through Data-Centric NLPcited by 17
  17. 2024MedQA-CS: Objective Structured Clinical Examination (OSCE)-Style Benchmark for Evaluating LLM Clinical Skillscited by 16
  18. 2024MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedbackcited by 14
  19. 2025S-Chain: Structured Visual Chain-of-Thought For Medicinecited by 10
  20. 2025DischargeSim: A Simulation Benchmark for Educational Doctor-Patient Communication at Dischargecited by 7
  21. 2025MedReadCtrl: Personalizing medical text generation with readability-controlled instruction learningcited by 5
  22. 2024MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosiscited by 4
  23. 2025Chatbot To Help Patients Understand Their Healthcited by 4
  24. 2025ChatThero: An LLM-Supported Chatbot for Behavior Change and Therapeutic Support in Addiction Recoverycited by 3
  25. 2026Medical thinking with multiple imagescited by 3
  26. 2025ChatCLIDS: Simulating Persuasive AI Dialogues to Promote Closed-Loop Insulin Adoption in Type 1 Diabetes Carecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MedQA-MM: Shortcuts Behind Medical Visual Reasoning
  2. 2025MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text
  3. 2025Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations
  4. 2023RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance
  5. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
  6. 2026IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations
  7. 2026Concept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generation
  8. 2025MedGEN-Bench: A Contextually Entangled Benchmark for Open-ended Multimodal Medical Generation