# Multimodal Fusion of EHR in Structures and Semantics: Integrating Clinical Records and Notes with Hypergraph and LLM

arXiv: https://arxiv.org/abs/2403.08818 · HTML: https://mlsanity.com/paper/2403.08818/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 15 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 7 of 7)

- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Hypergraph Neural Networks](https://mlsanity.com/paper/1809.09401/) (2018)
- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [Hypergraph Convolution and Hypergraph Attention](https://mlsanity.com/paper/1901.08150/) (2019)
- [Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis](https://mlsanity.com/paper/1706.03446/) (2017)
- [Learning the Graphical Structure of Electronic Health Records with Graph Convolutional Transformer](https://mlsanity.com/paper/1906.04716/) (2019)

## Built on by (most cited first, 10 of 10)

- [A Survey on Hypergraph Neural Networks: An In-Depth and Step-By-Step Guide](https://mlsanity.com/paper/2404.01039/) (2024)
- [Advancing Healthcare Automation: Multi-Agent System for Medical Necessity Justification](https://mlsanity.com/paper/2404.17977/) (2024)
- [Digital Diagnostics: The Potential Of Large Language Models In Recognizing Symptoms Of Common Illnesses](https://mlsanity.com/paper/2405.06712/) (2024)
- [Global Contrastive Training for Multimodal Electronic Health Records with Language Supervision](https://mlsanity.com/paper/2404.06723/) (2024)
- [Equitable Electronic Health Record Prediction with FAME: Fairness-Aware Multimodal Embedding](https://mlsanity.com/paper/2506.13104/) (2025)
- [Improving Hospital Risk Prediction with Knowledge-Augmented Multimodal EHR Modeling](https://mlsanity.com/paper/2508.01970/) (2025)
- [Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction](https://mlsanity.com/paper/2607.09982/) (2026)
- [Digital Twin Ecosystem for Oncology Clinical Operations](https://mlsanity.com/paper/2409.17650/) (2024)
- [Is Clinical Text Enough? A Multimodal Study on Mortality Prediction in Heart Failure Patients](https://mlsanity.com/paper/2604.01924/) (2026)
- [Large Language Models as Unified Multimodal Learners for Clinical Prediction](https://mlsanity.com/paper/2607.15380/) (2026)

## Abstract (arXiv)

Electronic Health Records (EHRs) have become increasingly popular to support clinical decision-making and healthcare in recent decades. EHRs usually contain heterogeneous information, such as structural data in tabular form and unstructured data in textual notes. Different types of information in EHRs can complement each other and provide a more complete picture of the health status of a patient. While there has been a lot of research on representation learning of structured EHR data, the fusion of different types of EHR data (multimodal fusion) is not well studied. This is mostly because of the complex medical coding systems used and the noise and redundancy present in the written notes. In this work, we propose a new framework called MINGLE, which integrates both structures and semantics in EHR effectively. Our framework uses a two-level infusion strategy to combine medical concept semantics and clinical note semantics into hypergraph neural networks, which learn the complex interactions between different types of data to generate visit representations for downstream prediction. Experiment results on two EHR datasets, the public MIMIC-III and private CRADLE, show that MINGLE can effectively improve predictive performance by 11.83% relatively, enhancing semantic integration as well as multimodal fusion for structural and textual EHR data.