# A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction

arXiv: https://arxiv.org/abs/2208.10240 · HTML: https://mlsanity.com/paper/2208.10240/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 9 Aug 2022.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Deep Representation Learning of Patient Data from Electronic Health Records (EHR): A Systematic Review](https://mlsanity.com/paper/2010.02809/) (2020)
- [Using Clinical Notes with Time Series Data for ICU Management](https://mlsanity.com/paper/1909.09702/) (2019)
- [Benchmarking machine learning models on multi-centre eICU critical care dataset](https://mlsanity.com/paper/1910.00964/) (2019)
- [How to Leverage Multimodal EHR Data for Better Medical Predictions?](https://mlsanity.com/paper/2110.15763/) (2021)
- [Predicting Patient Outcomes with Graph Representation Learning](https://mlsanity.com/paper/2101.03940/) (2021)

## Built on by (most cited first, 30 of 48)

- [TrojVLM: Backdoor Attack Against Vision Language Models](https://mlsanity.com/paper/2409.19232/) (2024)
- [Attention-Enhancing Backdoor Attacks Against BERT-based Models](https://mlsanity.com/paper/2310.14480/) (2023)
- [Credit Card Fraud Detection Using Advanced Transformer Model](https://mlsanity.com/paper/2406.03733/) (2024)
- [Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attention](https://mlsanity.com/paper/2305.02504/) (2023)
- [Task-Agnostic Detector for Insertion-Based Backdoor Attacks](https://mlsanity.com/paper/2403.17155/) (2024)
- [TaskCLIP: Extend Large Vision-Language Model for Task Oriented Object Detection](https://mlsanity.com/paper/2403.08108/) (2024)
- [ImageFlowNet: Forecasting Multiscale Image-Level Trajectories of Disease Progression with Irregularly-Sampled Longitudinal Medical Images](https://mlsanity.com/paper/2406.14794/) (2024)
- [Can Pre-trained Language Models Understand Chinese Humor?](https://mlsanity.com/paper/2407.04105/) (2024)
- [An Open and Large-Scale Dataset for Multi-Modal Climate Change-aware Crop Yield Predictions](https://mlsanity.com/paper/2406.06081/) (2024)
- [Advanced Payment Security System:XGBoost, LightGBM and SMOTE Integrated](https://mlsanity.com/paper/2406.04658/) (2024)
- [Mapping New Realities: Ground Truth Image Creation with Pix2Pix Image-to-Image Translation](https://mlsanity.com/paper/2404.19265/) (2024)
- [Time Matters: Examine Temporal Effects on Biomedical Language Models](https://mlsanity.com/paper/2407.17638/) (2024)
- [MoE-Health: A Mixture of Experts Framework for Robust Multimodal Healthcare Prediction](https://mlsanity.com/paper/2508.21793/) (2025)
- [Longitudinal Flow Matching for Trajectory Modeling](https://mlsanity.com/paper/2510.03569/) (2025)
- [TA-RNN: an Attention-based Time-aware Recurrent Neural Network Architecture for Electronic Health Records](https://mlsanity.com/paper/2401.14694/) (2024)
- [Can Contrastive Learning Refine Embeddings](https://mlsanity.com/paper/2404.08701/) (2024)
- [BadCLM: Backdoor Attack in Clinical Language Models for Electronic Health Records](https://mlsanity.com/paper/2407.05213/) (2024)
- [Towards Robust Vision Transformer via Masked Adaptive Ensemble](https://mlsanity.com/paper/2407.15385/) (2024)
- [Examining Imbalance Effects on Performance and Demographic Fairness of Clinical Language Models](https://mlsanity.com/paper/2412.17803/) (2024)
- [MIMONet: Multi-Input Multi-Output On-Device Deep Learning](https://mlsanity.com/paper/2307.11962/) (2023)
- [Temporal Cross-Attention for Dynamic Embedding and Tokenization of Multimodal Electronic Health Records](https://mlsanity.com/paper/2403.04012/) (2024)
- [SoK: Security and Privacy Risks of Healthcare AI](https://mlsanity.com/paper/2409.07415/) (2024)
- [Equitable Electronic Health Record Prediction with FAME: Fairness-Aware Multimodal Embedding](https://mlsanity.com/paper/2506.13104/) (2025)
- [MedPatch: Confidence-Guided Multi-Stage Fusion for Multimodal Clinical Data](https://mlsanity.com/paper/2508.09182/) (2025)
- [OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Prediction](https://mlsanity.com/paper/2604.16878/) (2026)
- [Real-Time Summarization of Twitter](https://mlsanity.com/paper/2407.08125/) (2024)
- [SAFER: A Calibrated Risk-Aware Multimodal Recommendation Model for Dynamic Treatment Regimes](https://mlsanity.com/paper/2506.06649/) (2025)
- [Moir'eXNet: Adaptive Multi-Scale Demoir'eing with Linear Attention Test-Time Training and Truncated Flow Matching Prior](https://mlsanity.com/paper/2506.15929/) (2025)
- [ClinStructor: AI-Powered Structuring of Unstructured Clinical Texts](https://mlsanity.com/paper/2511.11883/) (2025)
- [Transparent Early ICU Mortality Prediction with Clinical Transformer and Per-Case Modality Attribution](https://mlsanity.com/paper/2511.15847/) (2025)

## Abstract (arXiv)

Deep-learning-based clinical decision support using structured electronic health records (EHR) has been an active research area for predicting risks of mortality and diseases. Meanwhile, large amounts of narrative clinical notes provide complementary information, but are often not integrated into predictive models. In this paper, we provide a novel multimodal transformer to fuse clinical notes and structured EHR data for better prediction of in-hospital mortality. To improve interpretability, we propose an integrated gradients (IG) method to select important words in clinical notes and discover the critical structured EHR features with Shapley values. These important words and clinical features are visualized to assist with interpretation of the prediction outcomes. We also investigate the significance of domain adaptive pretraining and task adaptive fine-tuning on the Clinical BERT, which is used to learn the representations of clinical notes. Experiments demonstrated that our model outperforms other methods (AUCPR: 0.538, AUCROC: 0.877, F1:0.490).