mlsanityData to 14 Oct 2026

Integrated multimodal artificial intelligence framework for healthcare applications

arXiv 2202.12998 ↗First posted 25 Feb 2022cs.LG cs.AI cs.CYcited by 71

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence (AI) systems hold great promise to improve healthcare over the next decades. Specifically, AI systems leveraging multiple data sources and input modalities are poised to become a viable method to deliver more accurate results and deployable pipelines across a wide range of applications. In this work, we propose and evaluate a unified Holistic AI in Medicine (HAIM) framework to facilitate the generation and testing of AI systems that leverage multimodal inputs. Our approach uses generalizable data pre-processing and machine learning modeling stages that can be readily adapted for research and deployment in healthcare environments. We evaluate our HAIM framework by training and characterizing 14,324 independent models based on HAIM-MIMIC-MM, a multimodal clinical database (N=34,537 samples) containing 7,279 unique hospitalizations and 6,485 patients, spanning all possible input combinations of 4 data modalities (i.e., tabular, time-series, text, and images), 11 unique data sources and 12 predictive tasks. We show that this framework can consistently and robustly produce models that outperform similar single-source approaches across various healthcare demonstrations (by 6-33%), including 10 distinct chest pathology diagnoses, along with length-of-stay and 48-hour mortality predictions. We also quantify the contribution of each modality and data source using Shapley values, which demonstrates the heterogeneity in data modality importance and the necessity of multimodal inputs across different healthcare-relevant tasks. The generalizable properties and flexibility of our Holistic AI in Medicine (HAIM) framework could offer a promising pathway for future multimodal predictive systems in clinical and operational healthcare settings.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Builds on 11 of its 59 references that are in the map. First posted on arXiv 25 Feb 2022.

Builds on

11 of 59 references.

References · 11

  1. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  2. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  3. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  4. 2018Scalable and accurate deep learning for electronic health recordscited by 280
  5. 2021TorchXRayVision: A library of chest X-ray datasets and modelscited by 131
  6. 2019A deep learning system for differential diagnosis of skin diseasescited by 118
  7. 2019Multimodal Machine Learning-based Knee Osteoarthritis Progression Prediction from Plain Radiographs and Clinical Datacited by 23
  8. 2021MIMIC-IF: Interpretability and Fairness Evaluation of Deep Learning Models on MIMIC-IV Datasetcited by 15
  9. 2018Improving Hospital Mortality Prediction with Medical Named Entities and Multimodal Learningcited by 14
  10. 2020Improving Clinical Outcome Predictions Using Convolution over Medical Entities with Multimodal Learningcited by 11
  11. 2020Multimodal Learning for Cardiovascular Risk Prediction using EHR Datacited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 24; 2025: 19; 2026: 17.

71 papers.

Sort:

By area:

  1. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 68
  2. 2024Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRscited by 39
  3. 2024Review of multimodal machine learning approaches in healthcarecited by 31
  4. 2023The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRscited by 29
  5. 2023Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attentioncited by 25
  6. 2023Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicinecited by 24
  7. 2023MultiModN- Multimodal, Multi-Task, Interpretable Modular Networkscited by 14
  8. 2024When Raw Data Prevails: Are Large Language Model Embeddings Effective in Numerical Data Representation for Medical Machine Learning Applications?cited by 12
  9. 2025UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysiscited by 12
  10. 2023Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sourcescited by 10
  11. 2024Multimodal Data Integration for Precision Oncology: Challenges and Future Directionscited by 10
  12. 2024UrFound: Towards Universal Retinal Foundation Models via Knowledge-Guided Masked Modelingcited by 10
38 more
  1. 2022Unsupervised pre-training of graph transformers on patient population graphscited by 9
  2. 2023Harmonic-NAS: Hardware-Aware Multimodal Neural Architecture Search on Resource-constrained Devicescited by 8
  3. 2024Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learningcited by 8
  4. 2025Time Series Embedding Methods for Classification Tasks: A Reviewcited by 7
  5. 2025MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modellingcited by 7
  6. 2023MuG: A Multimodal Classification Benchmark on Game Data with Tabular, Textual, and Visual Fieldscited by 6
  7. 2024HoneyBee: A Scalable Modular Framework for Creating Multimodal Oncology Datasets with Foundational Embedding Modelscited by 6
  8. 2024FlexCare: Leveraging Cross-Task Synergy for Flexible Multimodal Healthcare Predictioncited by 6
  9. 2024Understanding Graphical Perception in Data Visualization through Zero-shot Prompting of Vision-Language Modelscited by 6
  10. 2025Cohort-Based Active Modality Acquisitioncited by 5
  11. 2025Fusion or Confusion? Multimodal Complexity Is Not All You Needcited by 5
  12. 2023Privacy-preserving machine learning for healthcare: open challenges and future perspectivescited by 4
  13. 2024HIST-AID: Leveraging Historical Patient Reports for Enhanced Multi-Modal Automatic Diagnosiscited by 4
  14. 2024Towards Foundation Models for Critical Care Time Seriescited by 4
  15. 2025Massively Multimodal Foundation Models: A Framework for Capturing Interactions with Specialized Mixture-of-Expertscited by 4
  16. 2025Reflections from Research Roundtables at the Conference on Health, Inference, and Learning (CHIL) 2025cited by 4
  17. 2024MedPromptX: Grounded Multimodal Prompting for Chest X-ray Diagnosiscited by 3
  18. 2024Automated Ensemble Multimodal Machine Learning for Healthcarecited by 3
  19. 2025PiCME: Pipeline for Contrastive Modality Evaluation and Encoding in the MIMIC Datasetcited by 3
  20. 2026A Multimodal Data Processing Pipeline for MIMIC-IV Datasetcited by 3
  21. 2023Ensemble Modeling for Time Series Forecasting: an Adaptive Robust Optimization Approachcited by 2
  22. 2024Interpretable Multimodal Learning for Cardiovascular Hemodynamics Assessmentcited by 2
  23. 2024Embedding-based Multimodal Learning on Pan-Squamous Cell Carcinomas for Improved Survival Outcomescited by 2
  24. 2024Predicting the Stay Length of Patients in Hospitals using Convolutional Gated Recurrent Deep Learning Modelcited by 2
  25. 2025Explainable Multimodal Regression via Information Decompositioncited by 2
  26. 2026TRACE: A Temporal Conditional Estimation for Multimodal Time Series Foundation Modelscited by 2
  27. 2024Policy Trees for Prediction: Interpretable and Adaptive Model Selection for Machine Learningcited by 1
  28. 2025VisTA: Vision-Text Alignment Model with Contrastive Learning using Multimodal Data for Evidence-Driven, Reliable, and Explainable Alzheimer's Disease Diagnosiscited by 1
  29. 2025Efficient Domain Adaptation of Multimodal Embeddings using Constrastive Learningcited by 1
  30. 2025Prediction of Significant Creatinine Elevation in First ICU Stays with Vancomycin Use: A retrospective study through Catboostcited by 1
  31. 2025A Framework for the Adoption and Integration of Generative AI in Midsize Organizations and Enterprises (FAIGMOE)cited by 1
  32. 2025Multi-Modal AI for Remote Patient Monitoring in Cancer Carecited by 1
  33. 2026Handling Missing Modalities in Multimodal Survival Prediction for Non-Small Cell Lung Cancercited by 1
  34. 2026Handling and Interpreting Missing Modalities in Patient Clinical Trajectories via Autoregressive Sequence Modelingcited by 1
  35. 2023On Preserving the Knowledge of Long Clinical Textscited by 0
  36. 2024Detecting Brain Tumors through Multimodal Neural Networkscited by 0
  37. 2024Foundation Model Makes Clustering A Better Initialization For Cold-Start Active Learningcited by 0
  38. 2024Towards Precision Healthcare: Robust Fusion of Time Series and Image Datacited by 0

50 most cited of 71 All 71 (JSON)