mlsanityData to 14 Oct 2026

Review of multimodal machine learning approaches in healthcare

arXiv 2402.02460 ↗First posted 4 Feb 2024cs.LG cs.AI cs.CVcited by 31

Not on the map. Why?

Abstract (arXiv)

Machine learning methods in healthcare have traditionally focused on using data from a single modality, limiting their ability to effectively replicate the clinical practice of integrating multiple sources of information for improved decision making. Clinicians typically rely on a variety of data sources including patients' demographic information, laboratory data, vital signs and various imaging data modalities to make informed decisions and contextualise their findings. Recent advances in machine learning have facilitated the more efficient incorporation of multimodal data, resulting in applications that better represent the clinician's approach. Here, we provide a review of multimodal machine learning approaches in healthcare, offering a comprehensive overview of recent literature. We discuss the various data modalities used in clinical diagnosis, with a particular emphasis on imaging data. We evaluate fusion techniques, explore existing multimodal datasets and examine common training strategies.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 68 of its 248 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

68 of 248 references.

References · 68

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  10. 2017A Survey on Deep Learning in Medical Image Analysiscited by 1,186
  11. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  12. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
38 more
  1. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 819
  2. 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
  3. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  4. 2020The Future of Digital Health with Federated Learningcited by 633
  5. 2021ImageNet-21K Pretraining for the Massescited by 613
  6. 2016Recurrent Neural Networks for Multivariate Time Series with Missing Valuescited by 554
  7. 2021Tabular Data: Deep Learning is Not All You Needcited by 535
  8. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  9. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  10. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  11. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 309
  12. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  13. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  14. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  15. 2019Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosiscited by 165
  16. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  17. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  18. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 148
  19. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  20. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  21. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  22. 2018Disease Prediction using Graph Convolutional Networks: Application to Autism Spectrum Disorder and Alzheimer's Diseasecited by 99
  23. 2021SiT: Self-supervised vIsion Transformercited by 99
  24. 2021Deep Learning for Chest X-ray Analysis: A Surveycited by 84
  25. 2020Secure and Robust Machine Learning for Healthcare: A Surveycited by 80
  26. 2021Self-supervised learning methods and applications in medical imaging analysis: A surveycited by 77
  27. 2022Integrated multimodal artificial intelligence framework for healthcare applicationscited by 71
  28. 2018Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classificationcited by 57
  29. 2019Multimodal Self-Supervised Learning for Medical Image Analysiscited by 57
  30. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  31. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  32. 2019Skin Lesion Classification Using Ensembles of Multi-Resolution EfficientNets with Meta Datacited by 43
  33. 2022Robust and Efficient Medical Imaging with Self-Supervisioncited by 40
  34. 2021Combining 3D Image and Tabular Data via the Dynamic Affine Feature Map Transformcited by 38
  35. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  36. 2022Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Reviewcited by 37
  37. 2019Natural Language Processing of Clinical Notes on Chronic Diseases: Systematic Reviewcited by 36
  38. 2021RadFusion: Benchmarking Performance and Fairness for Multimodal Pulmonary Embolism Detection from CT and EHRcited by 32

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 9; 2026: 13.

31 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  2. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  3. 2024Promoting cross-modal representations to improve multimodal foundation models for physiological signalscited by 13
  4. 2024Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applicationscited by 13
  5. 2025MoE-Health: A Mixture of Experts Framework for Robust Multimodal Healthcare Predictioncited by 7
  6. 2024LUMA: A Benchmark Dataset for Learning from Uncertain and Multimodal Datacited by 5
  7. 2024Electrocardiogram-Language Model for Few-Shot Question Answering with Meta Learningcited by 5
  8. 2024Multimodal Learning with Uncertainty Quantification based on Discounted Belief Fusioncited by 5
  9. 2025Reflections from Research Roundtables at the Conference on Health, Inference, and Learning (CHIL) 2025cited by 4
  10. 2025Towards deployment-centric multimodal AI beyond vision and languagecited by 3
  11. 2025The Moon's Many Faces: A Single Unified Transformer for Multimodal Lunar Reconstructioncited by 3
  12. 2025MedPatch: Confidence-Guided Multi-Stage Fusion for Multimodal Clinical Datacited by 2
19 more
  1. 2025CARDIUM: Congenital Anomaly Recognition with Diagnostic Images and Unified Medical recordscited by 2
  2. 2024More is Less? A Simulation-Based Approach to Dynamic Interactions between Biases in Multimodal Modelscited by 1
  3. 2025NAPS: Attention-Based Fusion of Heterogeneous Physiological Signalscited by 1
  4. 2026Med-MMFL: A Multimodal Federated Learning Benchmark in Healthcarecited by 1
  5. 2026Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attentioncited by 1
  6. 2026From pre-training to downstream performance: Does domain-specific pre-training make sense?cited by 1
  7. 2026Document Classification Pattern Recognition via Information Fusion: A Systematic Review of Multimodal and Multiview Representation Approachescited by 1
  8. 2024Towards Holistic Disease Risk Prediction using Small Language Modelscited by 0
  9. 2025Robust Multi-Modal Forecasting: Integrating Static and Dynamic Featurescited by 0
  10. 2025Few-Shot Multimodal Medical Imaging: A Theoretical Frameworkcited by 0
  11. 2026Multimodal Survival Modeling and Fairness-Aware Clinical Machine Learning for 5-Year Breast Cancer Risk Predictioncited by 0
  12. 2026Causal Transfer in Medical Image Analysiscited by 0
  13. 2026Out of Context: Reliability in Multimodal Anomaly Detection Requires Contextual Inferencecited by 0
  14. 2026MIRAGE: Retrieval and Generation of Multimodal Images and Texts for Medical Educationcited by 0
  15. 2026Multimodality Stacking with Blockwise missing values and application to the PIONeeR biomarkers study for prediction of resistance to immunotherapycited by 0
  16. 2026RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalitiescited by 0
  17. 2026Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysiscited by 0
  18. 2026Multimodal Cultural Heritage Architectural Style Classification for Residential Buildings in the UAE Based on CLIP Embeddings and SVMcited by 0
  19. 2026Design Choices for Multimodal Transformer Learning with Missing and Longitudinal Healthcare Datacited by 0