# An explainable Transformer-based deep learning model for the prediction of incident heart failure

arXiv: https://arxiv.org/abs/2101.11359 · HTML: https://mlsanity.com/paper/2101.11359/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 45 references that are in the map.
- First posted on arXiv 27 Jan 2021.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records](https://mlsanity.com/paper/1805.10724/) (2018)
- [Deepr: A Convolutional Net for Medical Records](https://mlsanity.com/paper/1607.07519/) (2016)

## Built on by (most cited first, 12 of 12)

- [Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records](https://mlsanity.com/paper/2106.11360/) (2021)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [Targeted-BEHRT: Deep learning for observational causal inference on longitudinal electronic health records](https://mlsanity.com/paper/2202.03487/) (2022)
- [Transformer-based Time-to-Event Prediction for Chronic Kidney Disease Deterioration](https://mlsanity.com/paper/2306.05779/) (2023)
- [Automatic Differential Diagnosis using Transformer-Based Multi-Label Sequence Classification](https://mlsanity.com/paper/2408.15827/) (2024)
- [Interpreting deep embeddings for disease progression clustering](https://mlsanity.com/paper/2307.06060/) (2023)
- [CARNA: Characterizing Advanced heart failure Risk and hemodyNAmic phenotypes using learned multi-valued decision diagrams](https://mlsanity.com/paper/2306.06801/) (2023)
- [Consensus of state of the art mortality prediction models: From all-cause mortality to sudden death prediction](https://mlsanity.com/paper/2308.16067/) (2023)
- [Deep multimodal saliency parcellation of cerebellar pathways: linking microstructure and individual function through explainable multitask learning](https://mlsanity.com/paper/2407.15132/) (2024)
- [A Laplace diffusion-based transformer model for heart rate forecasting within daily activity context](https://mlsanity.com/paper/2508.16655/) (2025)
- [Application of Graph Based Vision Transformers Architectures for Accurate Temperature Prediction in Fiber Specklegram Sensors](https://mlsanity.com/paper/2511.14792/) (2025)
- [Explainable Transformer Models for Clinical Prediction Tasks on Structured Electronic Health Records](https://mlsanity.com/paper/2608.20315/) (2026)

## Abstract (arXiv)

Predicting the incidence of complex chronic conditions such as heart failure is challenging. Deep learning models applied to rich electronic health records may improve prediction but remain unexplainable hampering their wider use in medical practice. We developed a novel Transformer deep-learning model for more accurate and yet explainable prediction of incident heart failure involving 100,071 patients from longitudinal linked electronic health records across the UK. On internal 5-fold cross validation and held-out external validation, our model achieved 0.93 and 0.93 area under the receiver operator curve and 0.69 and 0.70 area under the precision-recall curve, respectively and outperformed existing deep learning models. Predictor groups included all community and hospital diagnoses and medications contextualised within the age and calendar year for each patient's clinical encounter. The importance of contextualised medical information was revealed in a number of sensitivity analyses, and our perturbation method provided a way of identifying factors contributing to risk. Many of the identified risk factors were consistent with existing knowledge from clinical and epidemiological research but several new associations were revealed which had not been considered in expert-driven risk prediction models.