# EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generation

arXiv: https://arxiv.org/abs/2406.00036 · HTML: https://mlsanity.com/paper/2406.00036/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 22 of its 52 references that are in the map.
- First posted on arXiv 27 May 2024.

## Placed in

- Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/)

## Builds on (most cited first, 22 of 22)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Scalable and accurate deep learning for electronic health records](https://mlsanity.com/paper/1801.07860/) (2018)
- [Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?](https://mlsanity.com/paper/2308.10168/) (2023)
- [GRAM: Graph-based Attention Model for Healthcare Representation Learning](https://mlsanity.com/paper/1611.07012/) (2016)
- [Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate](https://mlsanity.com/paper/2305.13160/) (2023)
- [M3Care: Learning with Missing Modalities in Multimodal Healthcare Data](https://mlsanity.com/paper/2210.17292/) (2022)
- [GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphs](https://mlsanity.com/paper/2305.12788/) (2023)
- [A Closer Look at AUROC and AUPRC under Class Imbalance](https://mlsanity.com/paper/2401.06091/) (2024)
- [Using Clinical Notes with Time Series Data for ICU Management](https://mlsanity.com/paper/1909.09702/) (2019)
- [A Comparative Study of Pretrained Language Models for Long Clinical Text](https://mlsanity.com/paper/2301.11847/) (2023)
- [Collaborative Graph Learning with Auxiliary Text for Temporal Event Prediction in Healthcare](https://mlsanity.com/paper/2105.07542/) (2021)
- [How to Leverage Multimodal EHR Data for Better Medical Predictions?](https://mlsanity.com/paper/2110.15763/) (2021)
- [Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attention](https://mlsanity.com/paper/2305.02504/) (2023)
- [GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Ranking](https://mlsanity.com/paper/2305.18144/) (2023)
- [Recent Advances in Predictive Modeling with Electronic Health Records](https://mlsanity.com/paper/2402.01077/) (2024)
- [Graph-Text Multi-Modal Pre-training for Medical Representation Learning](https://mlsanity.com/paper/2203.09994/) (2022)
- [A Comprehensive Benchmark for COVID-19 Predictive Modeling Using Electronic Health Records in Intensive Care](https://mlsanity.com/paper/2209.07805/) (2022)
- [MedML: Fusing Medical Knowledge and Machine Learning Models for Early Pediatric COVID-19 Hospitalization and Severity Prediction](https://mlsanity.com/paper/2207.12283/) (2022)

## Built on by (most cited first, 17 of 17)

- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval](https://mlsanity.com/paper/2410.04585/) (2024)
- [EHR-RAG: Bridging Long-Horizon Structured Electronic Health Records and Large Language Models via Enhanced Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.21340/) (2026)
- [Multi-Ontology Integration with Dual-Axis Propagation for Medical Concept Representation](https://mlsanity.com/paper/2508.21320/) (2025)
- [Toward Better EHR Reasoning in LLMs: Reinforcement Learning with Expert Attention Guidance](https://mlsanity.com/paper/2508.13579/) (2025)
- [Clinical Knowledge Graph Construction and Evaluation with Multi-LLMs via Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.01844/) (2026)
- [Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environment](https://mlsanity.com/paper/2506.10030/) (2025)
- [Refine Medical Diagnosis Using Generation Augmented Retrieval and Clinical Practice Guidelines](https://mlsanity.com/paper/2506.21615/) (2025)
- [Grounded by Experience: Generative Healthcare Prediction Augmented with Hierarchical Agentic Retrieval](https://mlsanity.com/paper/2511.13293/) (2025)
- [Augmenting Clinical Decision-Making with an Interactive and Interpretable AI Copilot: A Real-World User Study with Clinicians in Nephrology and Obstetrics](https://mlsanity.com/paper/2602.00726/) (2026)
- [Mitigating Clinician Information Overload: Generative AI for Integrated EHR and RPM Data Analysis](https://mlsanity.com/paper/2509.00073/) (2025)
- [ReToP: Learning to Rewrite Electronic Health Records for Clinical Prediction](https://mlsanity.com/paper/2601.19286/) (2026)
- [TSAssistant: A Human-in-the-Loop Agentic Framework for Automated Target Safety Assessment](https://mlsanity.com/paper/2604.23938/) (2026)
- [EHR-RAGp: Retrieval-Augmented Prototype-Guided Foundation Model for Electronic Health Records](https://mlsanity.com/paper/2605.12335/) (2026)
- [Trajectory-Aware Clinical Risk Prediction via Severity-Grounded Knowledge Graphs and Retrieval-Augmented Generation](https://mlsanity.com/paper/2607.18270/) (2026)
- [HERMES: Contrast-Aware Knowledge Graph Reasoning from Clinical Notes for Patient Outcome Prediction](https://mlsanity.com/paper/2609.20825/) (2026)
- [Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical Prognosis](https://mlsanity.com/paper/2610.13510/) (2026)

## Abstract (arXiv)

The integration of multimodal Electronic Health Records (EHR) data has significantly advanced clinical predictive capabilities. Existing models, which utilize clinical notes and multivariate time-series EHR data, often fall short of incorporating the necessary medical context for accurate clinical tasks, while previous approaches with knowledge graphs (KGs) primarily focus on structured knowledge extraction. In response, we propose EMERGE, a Retrieval-Augmented Generation (RAG) driven framework to enhance multimodal EHR predictive modeling. We extract entities from both time-series data and clinical notes by prompting Large Language Models (LLMs) and align them with professional PrimeKG, ensuring consistency. In addition to triplet relationships, we incorporate entities' definitions and descriptions for richer semantics. The extracted knowledge is then used to generate task-relevant summaries of patients' health statuses. Finally, we fuse the summary with other modalities using an adaptive multimodal fusion network with cross-attention. Extensive experiments on the MIMIC-III and MIMIC-IV datasets' in-hospital mortality and 30-day readmission tasks demonstrate the superior performance of the EMERGE framework over baseline models. Comprehensive ablation studies and analysis highlight the efficacy of each designed module and robustness to data sparsity. EMERGE contributes to refining the utilization of multimodal EHR data in healthcare, bridging the gap with nuanced medical contexts essential for informed clinical predictions. We have publicly released the code at https://github.com/yhzhu99/EMERGE.