# Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications

arXiv: https://arxiv.org/abs/2411.03782 · HTML: https://mlsanity.com/paper/2411.03782/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 128 references that are in the map.
- First posted on arXiv 6 Nov 2024.

## Builds on (most cited first, 18 of 18)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images](https://mlsanity.com/paper/2302.14042/) (2023)
- [ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics](https://mlsanity.com/paper/2111.13424/) (2021)
- [Multi-modal Graph Learning for Disease Prediction](https://mlsanity.com/paper/2203.05880/) (2022)
- [Review of multimodal machine learning approaches in healthcare](https://mlsanity.com/paper/2402.02460/) (2024)
- [Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data](https://mlsanity.com/paper/2107.00648/) (2021)
- [Survival Prediction of Brain Cancer with Incomplete Radiology, Pathology, Genomics, and Demographic Data](https://mlsanity.com/paper/2203.04419/) (2022)
- [Multimodal Attention-based Deep Learning for Alzheimer's Disease Diagnosis](https://mlsanity.com/paper/2206.08826/) (2022)
- [Cascaded Multi-Modal Mixing Transformers for Alzheimer's Disease Classification with Incomplete Data](https://mlsanity.com/paper/2210.00255/) (2022)
- [Multimodal Explainability via Latent Shift applied to COVID-19 stratification](https://mlsanity.com/paper/2212.14084/) (2022)
- [RadioPathomics: Multimodal Learning in Non-Small Cell Lung Cancer for Adaptive Radiotherapy](https://mlsanity.com/paper/2204.12423/) (2022)
- [Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting from Multimodal Data](https://mlsanity.com/paper/2210.13889/) (2022)
- [Deep Multi-path Network Integrating Incomplete Biomarker and Chest CT Data for Evaluating Lung Cancer Risk](https://mlsanity.com/paper/2010.09524/) (2020)
- [BIO-CXRNET: A Robust Multimodal Stacking Machine Learning Technique for Mortality Risk Prediction of COVID-19 Patients using Chest X-Ray Images and Clinical Data](https://mlsanity.com/paper/2206.07595/) (2022)
- [BioFusionNet: Deep Learning-Based Survival Risk Stratification in ER+ Breast Cancer Through Multifeature and Multimodal Data Fusion](https://mlsanity.com/paper/2402.10717/) (2024)
- [Integrative Analysis of Patient Health Records and Neuroimages via Memory-based Graph Convolutional Network](https://mlsanity.com/paper/1809.06018/) (2018)
- [MaxCorrMGNN: A Multi-Graph Neural Network Framework for Generalized Multimodal Fusion of Medical Data for Outcome Prediction](https://mlsanity.com/paper/2307.07093/) (2023)

## Built on by (most cited first, 13 of 13)

- [MIRAGE: The Illusion of Visual Understanding](https://mlsanity.com/paper/2603.21687/) (2026)
- [Advancing Conversational Diagnostic AI with Multimodal Reasoning](https://mlsanity.com/paper/2505.04653/) (2025)
- [Towards a Medical AI Scientist](https://mlsanity.com/paper/2603.28589/) (2026)
- [Mask of truth: model sensitivity to unexpected regions of medical images](https://mlsanity.com/paper/2412.04030/) (2024)
- [Comparison of ConvNeXt and Vision-Language Models for Breast Density Assessment in Screening Mammography](https://mlsanity.com/paper/2506.13964/) (2025)
- [OncoVision: Integrating Mammography and Clinical Data through Attention-Driven Multimodal AI for Enhanced Breast Cancer Diagnosis](https://mlsanity.com/paper/2511.19667/) (2025)
- [MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models](https://mlsanity.com/paper/2607.07673/) (2026)
- [EAGLE: Efficient Alignment of Generalized Latent Embeddings for Multimodal Survival Prediction with Interpretable Attribution Analysis](https://mlsanity.com/paper/2506.22446/) (2025)
- [Intelligent Healthcare Imaging Platform: A VLM-Based Framework for Automated Medical Image Analysis and Clinical Report Generation](https://mlsanity.com/paper/2509.13590/) (2025)
- [Interpretable Multimodal Cancer Prototyping with Whole Slide Images and Incompletely Paired Genomics](https://mlsanity.com/paper/2511.21937/) (2025)
- [Resilient Vision-Tabular Multimodal Learning under Modality Missingness](https://mlsanity.com/paper/2605.12031/) (2026)
- [CHIMERA Challenge: Biochemical Recurrence Prediction in Prostate Cancer Patients using multimodal datasets](https://mlsanity.com/paper/2608.21497/) (2026)
- [BioMed-Agent-RL: A Meta Learning, All You Need for Biomedical Applications](https://mlsanity.com/paper/2608.21864/) (2026)

## Abstract (arXiv)

Recent technological advances in healthcare have led to unprecedented growth in patient data quantity and diversity. While artificial intelligence (AI) models have shown promising results in analyzing individual data modalities, there is increasing recognition that models integrating multiple complementary data sources, so-called multimodal AI, could enhance clinical decision-making. This scoping review examines the landscape of deep learning-based multimodal AI applications across the medical domain, analyzing 432 papers published between 2018 and 2024. We provide an extensive overview of multimodal AI development across different medical disciplines, examining various architectural approaches, fusion strategies, and common application areas. Our analysis reveals that multimodal AI models consistently outperform their unimodal counterparts, with an average improvement of 6.2 percentage points in AUC. However, several challenges persist, including cross-departmental coordination, heterogeneous data characteristics, and incomplete datasets. We critically assess the technical and practical challenges in developing multimodal AI systems and discuss potential strategies for their clinical implementation, including a brief overview of commercially available multimodal AI models for clinical decision-making. Additionally, we identify key factors driving multimodal AI development and propose recommendations to accelerate the field's maturation. This review provides researchers and clinicians with a thorough understanding of the current state, challenges, and future directions of multimodal AI in medicine.