# Multimodal Machine Learning in Precision Health

arXiv: https://arxiv.org/abs/2204.04777 · HTML: https://mlsanity.com/paper/2204.04777/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 197 references that are in the map.
- First posted on arXiv 10 Apr 2022.

## Builds on (most cited first, 22 of 22)

- [BRITS: Bidirectional Recurrent Imputation for Time Series](https://mlsanity.com/paper/1805.10572/) (2018)
- [TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays](https://mlsanity.com/paper/1801.04334/) (2018)
- [Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond](https://mlsanity.com/paper/2102.01998/) (2021)
- [RAIM: Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring Data](https://mlsanity.com/paper/1807.08820/) (2018)
- [Multimodal Machine Learning for Automated ICD Coding](https://mlsanity.com/paper/1810.13348/) (2018)
- [A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports](https://mlsanity.com/paper/2009.01523/) (2020)
- [Multimodal Machine Learning-based Knee Osteoarthritis Progression Prediction from Plain Radiographs and Clinical Data](https://mlsanity.com/paper/1904.06236/) (2019)
- [Clinical Text Classification with Rule-based Features and Knowledge-guided Convolutional Neural Networks](https://mlsanity.com/paper/1807.07425/) (2018)
- [Natural Language Processing for EHR-Based Computational Phenotyping](https://mlsanity.com/paper/1806.04820/) (2018)
- [TandemNet: Distilling Knowledge from Medical Images Using Diagnostic Reports as Optional Semantic References](https://mlsanity.com/paper/1708.03070/) (2017)
- [Multimodal fusion of imaging and genomics for lung cancer recurrence prediction](https://mlsanity.com/paper/2002.01982/) (2020)
- [Improving Clinical Outcome Predictions Using Convolution over Medical Entities with Multimodal Learning](https://mlsanity.com/paper/2011.12349/) (2020)
- [Multimodal Spatio-Temporal Deep Learning Approach for Neonatal Postoperative Pain Assessment](https://mlsanity.com/paper/2012.02175/) (2020)
- [Multiple Kernel Learning in the Primal for Multi-modal Alzheimer's Disease Classification](https://mlsanity.com/paper/1310.0890/) (2013)
- [Radiological images and machine learning: trends, perspectives, and prospects](https://mlsanity.com/paper/1903.11726/) (2019)
- [Mixed Pooling Multi-View Attention Autoencoder for Representation Learning in Healthcare](https://mlsanity.com/paper/1910.06456/) (2019)
- [Integrating Hypertension Phenotype and Genotype with Hybrid Non-negative Matrix Factorization](https://mlsanity.com/paper/1805.05008/) (2018)
- [Mixture-based Multiple Imputation Model for Clinical Data with a Temporal Dimension](https://mlsanity.com/paper/1908.04209/) (2019)
- [Characterizing Design Patterns of EHR-Driven Phenotype Extraction Algorithms](https://mlsanity.com/paper/1811.06183/) (2018)
- [Early Prediction of Mortality in Critical Care Setting in Sepsis Patients Using Structured Features and Unstructured Clinical Notes](https://mlsanity.com/paper/2112.01230/) (2021)
- [Med2Meta: Learning Representations of Medical Concepts with Meta-Embeddings](https://mlsanity.com/paper/1912.03366/) (2019)
- [Mixture Model Framework for Traumatic Brain Injury Prognosis Using Heterogeneous Clinical and Outcome Data](https://mlsanity.com/paper/2012.12310/) (2020)

## Built on by (most cited first, 3 of 3)

- [Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources](https://mlsanity.com/paper/2305.06217/) (2023)
- [SHINE: SubHypergraph Inductive Neural nEtwork](https://mlsanity.com/paper/2210.07309/) (2022)
- [Localising the Seizure Onset Zone from Single-Pulse Electrical Stimulation Responses with a CNN Transformer](https://mlsanity.com/paper/2403.20324/) (2024)

## Abstract (arXiv)

As machine learning and artificial intelligence are more frequently being leveraged to tackle problems in the health sector, there has been increased interest in utilizing them in clinical decision-support. This has historically been the case in single modal data such as electronic health record data. Attempts to improve prediction and resemble the multimodal nature of clinical expert decision-making this has been met in the computational field of machine learning by a fusion of disparate data. This review was conducted to summarize this field and identify topics ripe for future research. We conducted this review in accordance with the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) extension for Scoping Reviews to characterize multi-modal data fusion in health. We used a combination of content analysis and literature searches to establish search strings and databases of PubMed, Google Scholar, and IEEEXplore from 2011 to 2021. A final set of 125 articles were included in the analysis. The most common health areas utilizing multi-modal methods were neurology and oncology. However, there exist a wide breadth of current applications. The most common form of information fusion was early fusion. Notably, there was an improvement in predictive performance performing heterogeneous data fusion. Lacking from the papers were clear clinical deployment strategies and pursuit of FDA-approved tools. These findings provide a map of the current literature on multimodal data fusion as applied to health diagnosis/prognosis problems. Multi-modal machine learning, while more robust in its estimations over unimodal methods, has drawbacks in its scalability and the time-consuming nature of information concatenation.