# BERTSurv: BERT-Based Survival Models for Predicting Outcomes of Trauma Patients

arXiv: https://arxiv.org/abs/2103.10928 · HTML: https://mlsanity.com/paper/2103.10928/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 30 references that are in the map.
- First posted on arXiv 19 Mar 2021.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [DeepSurv: Personalized Treatment Recommender System Using A Cox Proportional Hazards Deep Neural Network](https://mlsanity.com/paper/1606.00931/) (2016)
- [Pre-training of Graph Augmented Transformers for Medication Recommendation](https://mlsanity.com/paper/1906.00346/) (2019)

## Built on by (most cited first, 15 of 15)

- [SurvTRACE: Transformers for Survival Analysis with Competing Events](https://mlsanity.com/paper/2110.00855/) (2021)
- [Integrating Physiological Time Series and Clinical Notes with Transformer for Early Prediction of Sepsis](https://mlsanity.com/paper/2203.14469/) (2022)
- [What Do You See in this Patient? Behavioral Testing of Clinical NLP Models](https://mlsanity.com/paper/2111.15512/) (2021)
- [FairEHR-CLP: Towards Fairness-Aware Clinical Predictions with Contrastive Learning in Multimodal Electronic Health Records](https://mlsanity.com/paper/2402.00955/) (2024)
- [Unveiling and Mitigating Bias in Mental Health Analysis with Large Language Models](https://mlsanity.com/paper/2406.12033/) (2024)
- [Enhancing Transformer Efficiency for Multivariate Time Series Classification](https://mlsanity.com/paper/2203.14472/) (2022)
- [Predicting the Need for Blood Transfusion in Intensive Care Units with Reinforcement Learning](https://mlsanity.com/paper/2206.14198/) (2022)
- [Dynamic Survival Transformers for Causal Inference with Electronic Health Records](https://mlsanity.com/paper/2210.15417/) (2022)
- [Vishing-Tactics-Bench: Forecasting Exploitation Trajectories in Voice Phishing Calls](https://mlsanity.com/paper/2609.07151/) (2026)
- [Time-to-event modeling of subreddits transitions to r/SuicideWatch](https://mlsanity.com/paper/2302.06030/) (2023)
- [On Preserving the Knowledge of Long Clinical Texts](https://mlsanity.com/paper/2311.01571/) (2023)
- [XAI for In-hospital Mortality Prediction via Multimodal ICU Data](https://mlsanity.com/paper/2312.17624/) (2023)
- [SigBERT: Combining Narrative Medical Reports and Rough Path Signature Theory for Survival Risk Estimation in Oncology](https://mlsanity.com/paper/2507.22941/) (2025)
- [Multimodal Survival Analysis with Locally Deployable Large Language Models](https://mlsanity.com/paper/2603.22158/) (2026)
- [Isotonic Survival Regression: Calibrated Survival Distributions from Deep Cox Models](https://mlsanity.com/paper/2605.16571/) (2026)

## Abstract (arXiv)

Survival analysis is a technique to predict the times of specific outcomes, and is widely used in predicting the outcomes for intensive care unit (ICU) trauma patients. Recently, deep learning models have drawn increasing attention in healthcare. However, there is a lack of deep learning methods that can model the relationship between measurements, clinical notes and mortality outcomes. In this paper we introduce BERTSurv, a deep learning survival framework which applies Bidirectional Encoder Representations from Transformers (BERT) as a language representation model on unstructured clinical notes, for mortality prediction and survival analysis. We also incorporate clinical measurements in BERTSurv. With binary cross-entropy (BCE) loss, BERTSurv can predict mortality as a binary outcome (mortality prediction). With partial log-likelihood (PLL) loss, BERTSurv predicts the probability of mortality as a time-to-event outcome (survival analysis). We apply BERTSurv on Medical Information Mart for Intensive Care III (MIMIC III) trauma patient data. For mortality prediction, BERTSurv obtained an area under the curve of receiver operating characteristic curve (AUC-ROC) of 0.86, which is an improvement of 3.6% over baseline of multilayer perceptron (MLP) without notes. For survival analysis, BERTSurv achieved a concordance index (C-index) of 0.7. In addition, visualizations of BERT's attention heads help to extract patterns in clinical notes and improve model interpretability by showing how the model assigns weights to different inputs.