# Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology

arXiv: https://arxiv.org/abs/2402.14252 · HTML: https://mlsanity.com/paper/2402.14252/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2024.

## Built on by (most cited first, 30 of 45)

- [MAIRA-2: Grounded Radiology Report Generation](https://mlsanity.com/paper/2406.04449/) (2024)
- [A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges](https://mlsanity.com/paper/2501.02189/) (2025)
- [PadChest-GR: A Bilingual Chest X-ray Dataset for Grounded Radiology Report Generation](https://mlsanity.com/paper/2411.05085/) (2024)
- [An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation](https://mlsanity.com/paper/2410.03334/) (2024)
- [Comprehensive language-image pre-training for 3D medical image understanding](https://mlsanity.com/paper/2510.15042/) (2025)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [FactCheXcker: Mitigating Measurement Hallucinations in Chest X-ray Report Generation Models](https://mlsanity.com/paper/2411.18672/) (2024)
- [AI, Help Me Thinkunicodex2014but for Myself: Assisting People in Complex Decision-Making by Providing Different Kinds of Cognitive Support](https://mlsanity.com/paper/2504.06771/) (2025)
- [EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models](https://mlsanity.com/paper/2509.20146/) (2025)
- [Enhancing Human-Computer Interaction in Chest X-ray Analysis using Vision and Language Model with Eye Gaze Patterns](https://mlsanity.com/paper/2404.02370/) (2024)
- [From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice](https://mlsanity.com/paper/2410.01812/) (2024)
- [A Multimodal Multi-Agent Framework for Radiology Report Generation](https://mlsanity.com/paper/2505.09787/) (2025)
- [Visual Prompt Engineering for Vision Language Models in Radiology](https://mlsanity.com/paper/2408.15802/) (2024)
- [RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering](https://mlsanity.com/paper/2512.17396/) (2025)
- [NL-Eye: Abductive NLI for Images](https://mlsanity.com/paper/2410.02613/) (2024)
- [Survey of User Interface Design and Interaction Techniques in Generative AI Applications](https://mlsanity.com/paper/2410.22370/) (2024)
- [CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding](https://mlsanity.com/paper/2509.23379/) (2025)
- [A Reliable Framework for Human-in-the-Loop Anomaly Detection in Time Series](https://mlsanity.com/paper/2405.03234/) (2024)
- [AI-Enhanced Sensemaking: Exploring the Design of a Generative AI-Based Assistant to Support Genetic Professionals](https://mlsanity.com/paper/2412.15444/) (2024)
- [Keypoint-Integrated Instruction-Following Data Generation for Enhanced Human Pose and Action Understanding in Multimodal Models](https://mlsanity.com/paper/2409.09306/) (2024)
- [Towards Child-Inclusive Clinical Video Understanding for Autism Spectrum Disorder](https://mlsanity.com/paper/2409.13606/) (2024)
- [Augmenting Human Cognition With Generative AI: Lessons From AI-Assisted Decision-Making](https://mlsanity.com/paper/2504.03207/) (2025)
- [Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibration](https://mlsanity.com/paper/2506.22819/) (2025)
- [Insights into a radiology-specialised multimodal large language model with sparse autoencoders](https://mlsanity.com/paper/2507.12950/) (2025)
- [Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership Inference Privacy Leakage?](https://mlsanity.com/paper/2511.20710/) (2025)
- [Sycophancy Undermines Epistemic Vigilance in Cooperative Vision-Language Tasks](https://mlsanity.com/paper/2607.29585/) (2026)
- ["It depends": Configuring AI to Improve Clinical Usefulness Across Contexts](https://mlsanity.com/paper/2407.11978/) (2024)
- [Your voice is your voice: Supporting Self-expression through Speech Generation and LLMs in Augmented and Alternative Communication](https://mlsanity.com/paper/2503.17479/) (2025)
- [Adaptive Human-Agent Teaming: A Review of Empirical Studies from the Process Dynamics Perspective](https://mlsanity.com/paper/2504.10918/) (2025)
- [Position: Restructuring of Categories and Implementation of Guidelines Essential for VLM Adoption in Healthcare](https://mlsanity.com/paper/2505.08818/) (2025)

## Abstract (arXiv)

Recent advances in AI combine large language models (LLMs) with vision encoders that bring forward unprecedented technical capabilities to leverage for a wide range of healthcare applications. Focusing on the domain of radiology, vision-language models (VLMs) achieve good performance results for tasks such as generating radiology findings based on a patient's medical image, or answering visual questions (e.g., 'Where are the nodules in this chest X-ray?'). However, the clinical utility of potential applications of these capabilities is currently underexplored. We engaged in an iterative, multidisciplinary design process to envision clinically relevant VLM interactions, and co-designed four VLM use concepts: Draft Report Generation, Augmented Report Review, Visual Search and Querying, and Patient Imaging History Highlights. We studied these concepts with 13 radiologists and clinicians who assessed the VLM concepts as valuable, yet articulated many design considerations. Reflecting on our findings, we discuss implications for integrating VLM capabilities in radiology, and for healthcare AI more generally.