# MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images

arXiv: https://arxiv.org/abs/2207.07027 · HTML: https://mlsanity.com/paper/2207.07027/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 56 references that are in the map.
- First posted on arXiv 14 Jul 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis](https://mlsanity.com/paper/1706.03446/) (2017)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [Multimodal Transformer with Multi-View Visual Representation for Image Captioning](https://mlsanity.com/paper/1905.07841/) (2019)
- [MIRACLE: Causally-Aware Imputation via Learning Missing Data Mechanisms](https://mlsanity.com/paper/2111.03187/) (2021)
- [End-to-end Multi-modal Video Temporal Grounding](https://mlsanity.com/paper/2107.05624/) (2021)
- [Medical Image Fusion: A survey of the state of the art](https://mlsanity.com/paper/1401.0166/) (2013)
- [Combining 3D Image and Tabular Data via the Dynamic Affine Feature Map Transform](https://mlsanity.com/paper/2107.05990/) (2021)
- [A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports](https://mlsanity.com/paper/2009.01523/) (2020)
- [Synthesizing the Unseen for Zero-shot Object Detection](https://mlsanity.com/paper/2010.09425/) (2020)
- [Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2111.07180/) (2021)
- [A Multimodal Deep Learning Model for Cardiac Resynchronisation Therapy Response Prediction](https://mlsanity.com/paper/2107.10662/) (2021)
- [Multi-Label Generalized Zero Shot Learning for the Classification of Disease in Chest Radiographs](https://mlsanity.com/paper/2107.06563/) (2021)
- [Towards dynamic multi-modal phenotyping using chest radiographs and physiological data](https://mlsanity.com/paper/2111.02710/) (2021)

## Built on by (most cited first, 30 of 54)

- [Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attention](https://mlsanity.com/paper/2305.02504/) (2023)
- [A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications](https://mlsanity.com/paper/2408.02686/) (2024)
- [Time-IMM: A Dataset and Benchmark for Irregular Multimodal Multivariate Time Series](https://mlsanity.com/paper/2506.10412/) (2025)
- [LANISTR: Multimodal Learning from Structured and Unstructured Data](https://mlsanity.com/paper/2305.16556/) (2023)
- [RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction](https://mlsanity.com/paper/2410.05361/) (2024)
- [On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?](https://mlsanity.com/paper/2505.15425/) (2025)
- [MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modelling](https://mlsanity.com/paper/2508.05492/) (2025)
- [One-Versus-Others Attention: Scalable Multimodal Integration for Biomedical Data](https://mlsanity.com/paper/2307.05435/) (2023)
- [Informative Priors Improve the Reliability of Multimodal Clinical Data Classification](https://mlsanity.com/paper/2312.00794/) (2023)
- [Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generation](https://mlsanity.com/paper/2410.17918/) (2024)
- [Borrowing Treasures from Neighbors: In-Context Learning for Multimodal Learning with Missing Modalities and Data Scarcity](https://mlsanity.com/paper/2403.09428/) (2024)
- [Retrieval-augmented in-context learning for multimodal large language models in disease classification](https://mlsanity.com/paper/2505.02087/) (2025)
- [Multimodal Representation Learning of Cardiovascular Magnetic Resonance Imaging](https://mlsanity.com/paper/2304.07675/) (2023)
- [Rethinking Gating Mechanism in Sparse MoE: Handling Arbitrary Modality Inputs with Confidence-Guided Gate](https://mlsanity.com/paper/2505.19525/) (2025)
- [PiCME: Pipeline for Contrastive Modality Evaluation and Encoding in the MIMIC Dataset](https://mlsanity.com/paper/2507.03165/) (2025)
- [AgentRx: A Benchmark Study of LLM Agents for Multimodal Clinical Prediction Tasks](https://mlsanity.com/paper/2605.10286/) (2026)
- [Towards Predicting Temporal Changes in a Patient's Chest X-ray Images based on Electronic Health Records](https://mlsanity.com/paper/2409.07012/) (2024)
- [MIND: Modality-Informed Knowledge Distillation Framework for Multimodal Clinical Prediction Tasks](https://mlsanity.com/paper/2502.01158/) (2025)
- [MedPatch: Confidence-Guided Multi-Stage Fusion for Multimodal Clinical Data](https://mlsanity.com/paper/2508.09182/) (2025)
- [An Empirical Analysis of Calibration and Selective Prediction in Multimodal Clinical Condition Classification](https://mlsanity.com/paper/2603.02719/) (2026)
- [OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Prediction](https://mlsanity.com/paper/2604.16878/) (2026)
- [Few-shot Learning using Data Augmentation and Time-Frequency Transformation for Time Series Classification](https://mlsanity.com/paper/2311.03194/) (2023)
- [Beyond Images: An Integrative Multi-modal Approach to Chest X-Ray Report Generation](https://mlsanity.com/paper/2311.11090/) (2023)
- [Deep Learning-Based Noninvasive Screening of Type 2 Diabetes with Chest X-ray Images and Electronic Health Records](https://mlsanity.com/paper/2412.10955/) (2024)
- [Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion](https://mlsanity.com/paper/2504.03135/) (2025)
- [Interpretable Hybrid-Rule Temporal Point Processes](https://mlsanity.com/paper/2504.11344/) (2025)
- [Small Lesions-aware Bidirectional Multimodal Multiscale Fusion Network for Lung Disease Classification](https://mlsanity.com/paper/2508.04205/) (2025)
- [Eyes on the Image: Gaze Supervised Multimodal Learning for Chest X-ray Diagnosis and Report Generation](https://mlsanity.com/paper/2508.13068/) (2025)
- [RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis](https://mlsanity.com/paper/2509.19980/) (2025)
- [Multimodal Disease Progression Modeling via Spatiotemporal Disentanglement and Multiscale Alignment](https://mlsanity.com/paper/2510.11112/) (2025)

## Abstract (arXiv)

Multi-modal fusion approaches aim to integrate information from different data sources. Unlike natural datasets, such as in audio-visual applications, where samples consist of "paired" modalities, data in healthcare is often collected asynchronously. Hence, requiring the presence of all modalities for a given sample is not realistic for clinical tasks and significantly limits the size of the dataset during training. In this paper, we propose MedFuse, a conceptually simple yet promising LSTM-based fusion module that can accommodate uni-modal as well as multi-modal input. We evaluate the fusion method and introduce new benchmark results for in-hospital mortality prediction and phenotype classification, using clinical time-series data in the MIMIC-IV dataset and corresponding chest X-ray images in MIMIC-CXR. Compared to more complex multi-modal fusion strategies, MedFuse provides a performance improvement by a large margin on the fully paired test set. It also remains robust across the partially paired test set containing samples with missing chest X-ray images. We release our code for reproducibility and to enable the evaluation of competing models in the future.