# MEDFuse: Multimodal EHR Data Fusion with Masked Lab-Test Modeling and Large Language Models

arXiv: https://arxiv.org/abs/2407.12309 · HTML: https://mlsanity.com/paper/2407.12309/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 37 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Builds on (most cited first, 10 of 10)

- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks](https://mlsanity.com/paper/1706.05764/) (2017)
- [Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction](https://mlsanity.com/paper/2401.01646/) (2024)
- [Deep Short Text Classification with Knowledge Powered Attention](https://mlsanity.com/paper/1902.08050/) (2019)
- [Evaluation Metrics for Unsupervised Learning Algorithms](https://mlsanity.com/paper/1905.05667/) (2019)
- [ZeroPrompt: Streaming Acoustic Encoders are Zero-Shot Masked LMs](https://mlsanity.com/paper/2305.10649/) (2023)
- [Labrador: Exploring the Limits of Masked Language Modeling for Laboratory Data](https://mlsanity.com/paper/2312.11502/) (2023)
- [Improved Multimodal Fusion for Small Datasets with Auxiliary Supervision](https://mlsanity.com/paper/2304.00379/) (2023)
- [Large Language Multimodal Models for 5-Year Chronic Disease Cohort Prediction Using EHR Data](https://mlsanity.com/paper/2403.04785/) (2024)

## Built on by (most cited first, 3 of 3)

- [Improving Hospital Risk Prediction with Knowledge-Augmented Multimodal EHR Modeling](https://mlsanity.com/paper/2508.01970/) (2025)
- [CURENet: Combining Unified Representations for Efficient Chronic Disease Prediction](https://mlsanity.com/paper/2511.11423/) (2025)
- [A General Multimodal Probing Framework for Data Fusion and Prediction with Frozen Large Language Models: Evidence from Multimodal EHR Data](https://mlsanity.com/paper/2606.28798/) (2026)

## Abstract (arXiv)

Electronic health records (EHRs) are multimodal by nature, consisting of structured tabular features like lab tests and unstructured clinical notes. In real-life clinical practice, doctors use complementary multimodal EHR data sources to get a clearer picture of patients' health and support clinical decision-making. However, most EHR predictive models do not reflect these procedures, as they either focus on a single modality or overlook the inter-modality interactions/redundancy. In this work, we propose MEDFuse, a Multimodal EHR Data Fusion framework that incorporates masked lab-test modeling and large language models (LLMs) to effectively integrate structured and unstructured medical data. MEDFuse leverages multimodal embeddings extracted from two sources: LLMs fine-tuned on free clinical text and masked tabular transformers trained on structured lab test results. We design a disentangled transformer module, optimized by a mutual information loss to 1) decouple modality-specific and modality-shared information and 2) extract useful joint representation from the noise and redundancy present in clinical notes. Through comprehensive validation on the public MIMIC-III dataset and the in-house FEMH dataset, MEDFuse demonstrates great potential in advancing clinical predictions, achieving over 90% F1 score in the 10-disease multi-label classification task.