mlsanityData to 14 Oct 2026

Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications

arXiv 2411.03782 ↗First posted 6 Nov 2024cs.AI cs.CY cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Recent technological advances in healthcare have led to unprecedented growth in patient data quantity and diversity. While artificial intelligence (AI) models have shown promising results in analyzing individual data modalities, there is increasing recognition that models integrating multiple complementary data sources, so-called multimodal AI, could enhance clinical decision-making. This scoping review examines the landscape of deep learning-based multimodal AI applications across the medical domain, analyzing 432 papers published between 2018 and 2024. We provide an extensive overview of multimodal AI development across different medical disciplines, examining various architectural approaches, fusion strategies, and common application areas. Our analysis reveals that multimodal AI models consistently outperform their unimodal counterparts, with an average improvement of 6.2 percentage points in AUC. However, several challenges persist, including cross-departmental coordination, heterogeneous data characteristics, and incomplete datasets. We critically assess the technical and practical challenges in developing multimodal AI systems and discuss potential strategies for their clinical implementation, including a brief overview of commercially available multimodal AI models for clinical decision-making. Additionally, we identify key factors driving multimodal AI development and propose recommendations to accelerate the field's maturation. This review provides researchers and clinicians with a thorough understanding of the current state, challenges, and future directions of multimodal AI in medicine.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 18 of its 128 references that are in the map. First posted on arXiv 6 Nov 2024.

Builds on

18 of 128 references.

References · 18

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  4. 2021ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Geneticscited by 36
  5. 2022Multi-modal Graph Learning for Disease Predictioncited by 34
  6. 2024Review of multimodal machine learning approaches in healthcarecited by 31
  7. 2021Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Datacited by 29
  8. 2022Survival Prediction of Brain Cancer with Incomplete Radiology, Pathology, Genomics, and Demographic Datacited by 25
  9. 2022Multimodal Attention-based Deep Learning for Alzheimer's Disease Diagnosiscited by 24
  10. 2022Cascaded Multi-Modal Mixing Transformers for Alzheimer's Disease Classification with Incomplete Datacited by 17
  11. 2022Multimodal Explainability via Latent Shift applied to COVID-19 stratificationcited by 16
  12. 2022RadioPathomics: Multimodal Learning in Non-Small Cell Lung Cancer for Adaptive Radiotherapycited by 10
6 more
  1. 2022Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting from Multimodal Datacited by 10
  2. 2020Deep Multi-path Network Integrating Incomplete Biomarker and Chest CT Data for Evaluating Lung Cancer Riskcited by 5
  3. 2022BIO-CXRNET: A Robust Multimodal Stacking Machine Learning Technique for Mortality Risk Prediction of COVID-19 Patients using Chest X-Ray Images and Clinical Datacited by 4
  4. 2024BioFusionNet: Deep Learning-Based Survival Risk Stratification in ER+ Breast Cancer Through Multifeature and Multimodal Data Fusioncited by 3
  5. 2018Integrative Analysis of Patient Health Records and Neuroimages via Memory-based Graph Convolutional Networkcited by 2
  6. 2023MaxCorrMGNN: A Multi-Graph Neural Network Framework for Generalized Multimodal Fusion of Medical Data for Outcome Predictioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 6.

13 papers.

Sort:

By area:

  1. 2026MIRAGE: The Illusion of Visual Understandingcited by 65
  2. 2025Advancing Conversational Diagnostic AI with Multimodal Reasoningcited by 20
  3. 2026Towards a Medical AI Scientistcited by 7
  4. 2024Mask of truth: model sensitivity to unexpected regions of medical imagescited by 6
  5. 2025Comparison of ConvNeXt and Vision-Language Models for Breast Density Assessment in Screening Mammographycited by 1
  6. 2025OncoVision: Integrating Mammography and Clinical Data through Attention-Driven Multimodal AI for Enhanced Breast Cancer Diagnosiscited by 1
  7. 2026MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Modelscited by 1
  8. 2025EAGLE: Efficient Alignment of Generalized Latent Embeddings for Multimodal Survival Prediction with Interpretable Attribution Analysiscited by 0
  9. 2025Intelligent Healthcare Imaging Platform: A VLM-Based Framework for Automated Medical Image Analysis and Clinical Report Generationcited by 0
  10. 2025Interpretable Multimodal Cancer Prototyping with Whole Slide Images and Incompletely Paired Genomicscited by 0
  11. 2026Resilient Vision-Tabular Multimodal Learning under Modality Missingnesscited by 0
  12. 2026CHIMERA Challenge: Biochemical Recurrence Prediction in Prostate Cancer Patients using multimodal datasetscited by 0
1 more
  1. 2026BioMed-Agent-RL: A Meta Learning, All You Need for Biomedical Applicationscited by 0