mlsanityData to 7 Oct 2026

ExBEHRT: Extended Transformer for Electronic Health Records to Predict Disease Subtypes & Progressions

arXiv 2303.12364 ↗First posted 22 Mar 2023cs.LG cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

In this study, we introduce ExBEHRT, an extended version of BEHRT (BERT applied to electronic health records), and apply different algorithms to interpret its results. While BEHRT considers only diagnoses and patient age, we extend the feature space to several multimodal records, namely demographics, clinical characteristics, vital signs, smoking status, diagnoses, procedures, medications, and laboratory tests, by applying a novel method to unify the frequencies and temporal dimensions of the different features. We show that additional features significantly improve model performance for various downstream tasks in different diseases. To ensure robustness, we interpret model predictions using an adaptation of expected gradients, which has not been previously applied to transformers with EHR data and provides more granular interpretations than previous approaches such as feature and token importances. Furthermore, by clustering the model representations of oncology patients, we show that the model has an implicit understanding of the disease and is able to classify patients with the same cancer type into different risk groups. Given the additional features and interpretability, ExBEHRT can help make informed decisions about disease trajectories, diagnoses, and risk factors of various diseases.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 16 of its 30 references that are in the map. First posted on arXiv 22 Mar 2023.

Builds on

16 of 30 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  4. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  5. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  7. 2017Focal Loss for Dense Object Detectioncited by 749
  8. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 368
  9. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  10. 2019BEHRT: Transformer for Electronic Health Recordscited by 258
  11. 2015Doctor AI: Predicting Clinical Events via Recurrent Neural Networkscited by 205
  12. 2019Pre-training of Graph Augmented Transformers for Medication Recommendationcited by 124
4 more
  1. 2021CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction taskscited by 75
  2. 2021Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health recordscited by 53
  3. 2021AMMU : A Survey of Transformer-based Biomedical Pretrained Language Modelscited by 32
  4. 2022Targeted-BEHRT: Deep learning for observational causal inference on longitudinal electronic health recordscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 6; 2026: 3.

12 papers.

  1. 2024Emergency Department Decision Support using Clinical Pseudo-notescited by 22
  2. 2025Serialized EHR make for good text representationscited by 7
  3. 2025Memorize and Rank: Elevating Large Language Models for Clinical Diagnosis Predictioncited by 6
  4. 2025Improving Representation Learning of Complex Critical Care Data with ICU-BERTcited by 1
  5. 2023An Interpretable Deep-Learning Framework for Predicting Hospital Readmissions From Electronic Health Recordscited by 0
  6. 2024Enhancing Multimodal Medical Image Classification using Cross-Graph Modal Contrastive Learningcited by 0
  7. 2025Revealing Interconnections between Diseases: from Statistical Methods to Large Language Modelscited by 0
  8. 2025Trajectory-guided discharge stratification for heart failure using short-context electronic health record sequence modelingcited by 0
  9. 2025PULSE-ICU: A Pretrained Unified Long-Sequence Encoder for Multi-task Prediction in Intensive Care Unitscited by 0
  10. 2026NPCNet: Navigator-Driven Pseudo Text for Deep Clustering of Early Sepsis Phenotypingcited by 0
  11. 2026Hierarchical Modeling of ICD Codes in EHR Foundation Modelscited by 0
  12. 2026Explainable Transformer Models for Clinical Prediction Tasks on Structured Electronic Health Recordscited by 0