# Towards dynamic multi-modal phenotyping using chest radiographs and physiological data

arXiv: https://arxiv.org/abs/2111.02710 · HTML: https://mlsanity.com/paper/2111.02710/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 18 references that are in the map.
- First posted on arXiv 4 Nov 2021.

## Builds on (most cited first, 6 of 6)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [DeepMMSA: A Novel Multimodal Deep Learning Method for Non-small Cell Lung Cancer Survival Analysis](https://mlsanity.com/paper/2106.06744/) (2021)
- [A Multimodal Deep Learning Model for Cardiac Resynchronisation Therapy Response Prediction](https://mlsanity.com/paper/2107.10662/) (2021)

## Built on by (most cited first, 5 of 5)

- [MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images](https://mlsanity.com/paper/2207.07027/) (2022)
- [Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosis](https://mlsanity.com/paper/2212.09162/) (2022)
- [Transformer representation learning is necessary for dynamic multi-modal physiological data on small-cohort patients](https://mlsanity.com/paper/2504.04120/) (2025)
- [Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Process](https://mlsanity.com/paper/2510.20736/) (2025)
- [Cross-Modal Alignment via Variational Copula Modelling](https://mlsanity.com/paper/2511.03196/) (2025)

## Abstract (arXiv)

The healthcare domain is characterized by heterogeneous data modalities, such as imaging and physiological data. In practice, the variety of medical data assists clinicians in decision-making. However, most of the current state-of-the-art deep learning models solely rely upon carefully curated data of a single modality. In this paper, we propose a dynamic training approach to learn modality-specific data representations and to integrate auxiliary features, instead of solely relying on a single modality. Our preliminary experiments results for a patient phenotyping task using physiological data in MIMIC-IV & chest radiographs in the MIMIC- CXR dataset show that our proposed approach achieves the highest area under the receiver operating characteristic curve (AUROC) (0.764 AUROC) compared to the performance of the benchmark method in previous work, which only used physiological data (0.740 AUROC). For a set of five recurring or chronic diseases with periodic acute episodes, including cardiac dysrhythmia, conduction disorders, and congestive heart failure, the AUROC improves from 0.747 to 0.798. This illustrates the benefit of leveraging the chest imaging modality in the phenotyping task and highlights the potential of multi-modal learning in medical applications.