# Label Dependent Attention Model for Disease Risk Prediction Using Multimodal Electronic Health Records

arXiv: https://arxiv.org/abs/2201.06779 · HTML: https://mlsanity.com/paper/2201.06779/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 18 Jan 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [Doctor AI: Predicting Clinical Events via Recurrent Neural Networks](https://mlsanity.com/paper/1511.05942/) (2015)
- [Label-Embedding for Image Classification](https://mlsanity.com/paper/1503.08677/) (2015)
- [GRAM: Graph-based Attention Model for Healthcare Representation Learning](https://mlsanity.com/paper/1611.07012/) (2016)
- [Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks](https://mlsanity.com/paper/1706.05764/) (2017)
- [GAMENet: Graph Augmented MEmory Networks for Recommending Medication Combination](https://mlsanity.com/paper/1809.01852/) (2018)
- [Joint Embedding of Words and Labels for Text Classification](https://mlsanity.com/paper/1805.04174/) (2018)
- [RAIM: Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring Data](https://mlsanity.com/paper/1807.08820/) (2018)
- [Explainable artificial intelligence model to predict acute critical illness from electronic health records](https://mlsanity.com/paper/1912.01266/) (2019)
- [COMPOSE: Cross-Modal Pseudo-Siamese Network for Patient Trial Matching](https://mlsanity.com/paper/2006.08765/) (2020)
- [Boosting Deep Learning Risk Prediction with Generative Adversarial Networks for Electronic Health Records](https://mlsanity.com/paper/1709.01648/) (2017)
- [Multi-Task Label Embedding for Text Classification](https://mlsanity.com/paper/1710.07210/) (2017)
- [Exploiting Convolutional Neural Network for Risk Prediction with Medical Feature Embedding](https://mlsanity.com/paper/1701.07474/) (2017)

## Built on by (most cited first, 5 of 5)

- [Knowledge-Augmented Multimodal Clinical Rationale Generation for Disease Diagnosis with Small Language Models](https://mlsanity.com/paper/2411.07611/) (2024)
- [HoloDx: Knowledge- and Data-Driven Multimodal Diagnosis of Alzheimer's Disease](https://mlsanity.com/paper/2504.19075/) (2025)
- [ProMedTS: A Self-Supervised, Prompt-Guided Multimodal Approach for Integrating Medical Text and Time Series](https://mlsanity.com/paper/2502.13509/) (2025)
- [Multimodal Interpretable Data-Driven Models for Early Prediction of Antimicrobial Multidrug Resistance Using Multivariate Time-Series](https://mlsanity.com/paper/2402.06295/) (2024)
- [Towards Precision Healthcare: Robust Fusion of Time Series and Image Data](https://mlsanity.com/paper/2405.15442/) (2024)

## Abstract (arXiv)

Disease risk prediction has attracted increasing attention in the field of modern healthcare, especially with the latest advances in artificial intelligence (AI). Electronic health records (EHRs), which contain heterogeneous patient information, are widely used in disease risk prediction tasks. One challenge of applying AI models for risk prediction lies in generating interpretable evidence to support the prediction results while retaining the prediction ability. In order to address this problem, we propose the method of jointly embedding words and labels whereby attention modules learn the weights of words from medical notes according to their relevance to the names of risk prediction labels. This approach boosts interpretability by employing an attention mechanism and including the names of prediction tasks in the model. However, its application is only limited to the handling of textual inputs such as medical notes. In this paper, we propose a label dependent attention model LDAM to 1) improve the interpretability by exploiting Clinical-BERT (a biomedical language model pre-trained on a large clinical corpus) to encode biomedically meaningful features and labels jointly; 2) extend the idea of joint embedding to the processing of time-series data, and develop a multi-modal learning framework for integrating heterogeneous information from medical notes and time-series health status indicators. To demonstrate our method, we apply LDAM to the MIMIC-III dataset to predict different disease risks. We evaluate our method both quantitatively and qualitatively. Specifically, the predictive power of LDAM will be shown, and case studies will be carried out to illustrate its interpretability.