mlsanityData to 14 Oct 2026

A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction

arXiv 2208.10240 ↗First posted 9 Aug 2022cs.CLcited by 48

Not on the map. Why?

Abstract (arXiv)

Deep-learning-based clinical decision support using structured electronic health records (EHR) has been an active research area for predicting risks of mortality and diseases. Meanwhile, large amounts of narrative clinical notes provide complementary information, but are often not integrated into predictive models. In this paper, we provide a novel multimodal transformer to fuse clinical notes and structured EHR data for better prediction of in-hospital mortality. To improve interpretability, we propose an integrated gradients (IG) method to select important words in clinical notes and discover the critical structured EHR features with Shapley values. These important words and clinical features are visualized to assist with interpretation of the prediction outcomes. We also investigate the significance of domain adaptive pretraining and task adaptive fine-tuning on the Clinical BERT, which is used to learn the representations of clinical notes. Experiments demonstrated that our model outperforms other methods (AUCPR: 0.538, AUCROC: 0.877, F1:0.490).

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 17 of its 35 references that are in the map. First posted on arXiv 9 Aug 2022.

Builds on

17 of 35 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  8. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  9. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  10. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
  11. 2017Multitask learning and benchmarking with clinical time series datacited by 290
  12. 2019BEHRT: Transformer for Electronic Health Recordscited by 263
5 more
  1. 2020Deep Representation Learning of Patient Data from Electronic Health Records (EHR): A Systematic Reviewcited by 47
  2. 2019Using Clinical Notes with Time Series Data for ICU Managementcited by 38
  3. 2019Benchmarking machine learning models on multi-centre eICU critical care datasetcited by 35
  4. 2021How to Leverage Multimodal EHR Data for Better Medical Predictions?cited by 26
  5. 2021Predicting Patient Outcomes with Graph Representation Learningcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 19; 2025: 13; 2026: 11.

48 papers.

Sort:

By area:

  1. 2024TrojVLM: Backdoor Attack Against Vision Language Modelscited by 39
  2. 2023Attention-Enhancing Backdoor Attacks Against BERT-based Modelscited by 32
  3. 2024Credit Card Fraud Detection Using Advanced Transformer Modelcited by 26
  4. 2023Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attentioncited by 25
  5. 2024Task-Agnostic Detector for Insertion-Based Backdoor Attackscited by 23
  6. 2024TaskCLIP: Extend Large Vision-Language Model for Task Oriented Object Detectioncited by 21
  7. 2024ImageFlowNet: Forecasting Multiscale Image-Level Trajectories of Disease Progression with Irregularly-Sampled Longitudinal Medical Imagescited by 19
  8. 2024Can Pre-trained Language Models Understand Chinese Humor?cited by 13
  9. 2024An Open and Large-Scale Dataset for Multi-Modal Climate Change-aware Crop Yield Predictionscited by 11
  10. 2024Advanced Payment Security System:XGBoost, LightGBM and SMOTE Integratedcited by 8
  11. 2024Mapping New Realities: Ground Truth Image Creation with Pix2Pix Image-to-Image Translationcited by 7
  12. 2024Time Matters: Examine Temporal Effects on Biomedical Language Modelscited by 7
36 more
  1. 2025MoE-Health: A Mixture of Experts Framework for Robust Multimodal Healthcare Predictioncited by 7
  2. 2025Longitudinal Flow Matching for Trajectory Modelingcited by 7
  3. 2024TA-RNN: an Attention-based Time-aware Recurrent Neural Network Architecture for Electronic Health Recordscited by 6
  4. 2024Can Contrastive Learning Refine Embeddingscited by 5
  5. 2024BadCLM: Backdoor Attack in Clinical Language Models for Electronic Health Recordscited by 5
  6. 2024Towards Robust Vision Transformer via Masked Adaptive Ensemblecited by 4
  7. 2024Examining Imbalance Effects on Performance and Demographic Fairness of Clinical Language Modelscited by 4
  8. 2023MIMONet: Multi-Input Multi-Output On-Device Deep Learningcited by 3
  9. 2024Temporal Cross-Attention for Dynamic Embedding and Tokenization of Multimodal Electronic Health Recordscited by 3
  10. 2024SoK: Security and Privacy Risks of Healthcare AIcited by 2
  11. 2025Equitable Electronic Health Record Prediction with FAME: Fairness-Aware Multimodal Embeddingcited by 2
  12. 2025MedPatch: Confidence-Guided Multi-Stage Fusion for Multimodal Clinical Datacited by 2
  13. 2026OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Predictioncited by 2
  14. 2024Real-Time Summarization of Twittercited by 1
  15. 2025SAFER: A Calibrated Risk-Aware Multimodal Recommendation Model for Dynamic Treatment Regimescited by 1
  16. 2025Moir'eXNet: Adaptive Multi-Scale Demoir'eing with Linear Attention Test-Time Training and Truncated Flow Matching Priorcited by 1
  17. 2025ClinStructor: AI-Powered Structuring of Unstructured Clinical Textscited by 1
  18. 2025Transparent Early ICU Mortality Prediction with Clinical Transformer and Per-Case Modality Attributioncited by 1
  19. 2026Multimodal Routing for Interpretable, Robust, and Auditable Clinical Predictioncited by 1
  20. 2023Word Definitions from Large Language Modelscited by 0
  21. 2023XAI for In-hospital Mortality Prediction via Multimodal ICU Datacited by 0
  22. 2024Analyzing Multi-Head Attention on Trojan BERT Modelscited by 0
  23. 2025CAAT-EHR: Cross-Attentional Autoregressive Transformer for Multimodal Electronic Health Record Embeddingscited by 0
  24. 2025Assessing the Impact of the Quality of Textual Data on Feature Representation and Machine Learning Modelscited by 0
  25. 2025DENSE: Longitudinal Progress Note Generation with Temporal Modeling of Heterogeneous Clinical Notes Across Hospital Visitscited by 0
  26. 2025Learning to Route: Per-Sample Adaptive Routing for Multimodal Multitask Predictioncited by 0
  27. 2025Evaluating Strategies for Synthesizing Clinical Notes for Medical Multimodal AIcited by 0
  28. 2026Temporal Fusion Nexus: A task-agnostic multi-modal embedding model for clinical narratives and irregular time series in post-kidney transplant carecited by 0
  29. 2026Is Clinical Text Enough? A Multimodal Study on Mortality Prediction in Heart Failure Patientscited by 0
  30. 2026CLIC: Contextual Language-Informed Cardiac Pathology Classificationcited by 0
  31. 2026A General Multimodal Probing Framework for Data Fusion and Prediction with Frozen Large Language Models: Evidence from Multimodal EHR Datacited by 0
  32. 2026Trajectory-Aware Clinical Risk Prediction via Severity-Grounded Knowledge Graphs and Retrieval-Augmented Generationcited by 0
  33. 2026CRS-Triage: Confidence- and Reliability-Aware Selective Triage under Incomplete Clinical Evidencecited by 0
  34. 2026Removing Temporal Note Redundancy Improves Multimodal Reinforcement Learning for Medicinecited by 0
  35. 2026SHIFT-M3: Pre-fusion Alignment-based Consistency Screening for Multimodal ECG Record Integritycited by 0
  36. 2026MedGate-Fusion: Integrating First-Encounter Semantic Narratives and Physiological Biomarkers for Prospective Stroke Risk Stratificationcited by 0