mlsanityData to 14 Oct 2026

CORE-BEHRT: A Carefully Optimized and Rigorously Evaluated BEHRT

arXiv 2404.15201 ↗First posted 23 Apr 2024cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

The widespread adoption of Electronic Health Records (EHR) has significantly increased the amount of available healthcare data. This has allowed models inspired by Natural Language Processing (NLP) and Computer Vision, which scale exceptionally well, to be used in EHR research. Particularly, BERT-based models have surged in popularity following the release of BEHRT and Med-BERT. Subsequent models have largely built on these foundations despite the fundamental design choices of these pioneering models remaining underexplored. Through incremental optimization, we study BERT-based EHR modeling and isolate the sources of improvement for key design choices, giving us insights into the effect of data representation, individual technical components, and training procedure. Evaluating this across a set of generic tasks (death, pain treatment, and general infection), we showed that improving data representation can increase the average downstream performance from 0.785 to 0.797 AUROC (p<10-7), primarily when including medication and timestamps. Improving the architecture and training protocol on top of this increased average downstream performance to 0.801 AUROC (p<10-7). We then demonstrated the consistency of our optimization through a rigorous evaluation across 25 diverse clinical prediction tasks. We observed significant performance increases in 17 out of 25 tasks and improvements in 24 tasks, highlighting the generalizability of our results. Our findings provide a strong foundation for future work and aim to increase the trustworthiness of BERT-based EHR models.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 23 of its 48 references that are in the map. First posted on arXiv 23 Apr 2024.

Builds on

23 of 48 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2020GLU Variants Improve Transformercited by 1,804
  11. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  12. 2016Language Modeling with Gated Convolutional Networkscited by 702
11 more
  1. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  2. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 373
  3. 2016RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanismcited by 349
  4. 2019BEHRT: Transformer for Electronic Health Recordscited by 263
  5. 2019Time2Vec: Learning a Vector Representation of Timecited by 213
  6. 2015Doctor AI: Predicting Clinical Events via Recurrent Neural Networkscited by 208
  7. 2019Pre-training of Graph Augmented Transformers for Medication Recommendationcited by 127
  8. 2021CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction taskscited by 76
  9. 2016Deepr: A Convolutional Net for Medical Recordscited by 60
  10. 2021Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health recordscited by 54
  11. 2022Targeted-BEHRT: Deep learning for observational causal inference on longitudinal electronic health recordscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 13.

20 papers.

Sort:

By area:

  1. 2024Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRscited by 39
  2. 2025Large Language Models are Powerful Electronic Health Record Encoderscited by 26
  3. 2025A Collection of Innovations in Medical AI for patient records in 2024cited by 7
  4. 2025FoMoH: A clinically meaningful foundation model evaluation for structured electronic health recordscited by 6
  5. 2026One Loss to Rule Them All: Marked Time-to-Event for Structured EHR Foundation Modelscited by 5
  6. 2025CEHR-XGPT: A Scalable Multi-Task Foundation Model for Electronic Health Recordscited by 4
  7. 2026Retrieval-aligned Tabular Foundation Models Enable Robust Clinical Risk Prediction in Electronic Health Records Under Real-world Constraintscited by 4
  8. 2026Learning Clinical Representations Under Systematic Distribution Shiftcited by 2
  9. 2026Discriminative Representation Learning for Clinical Predictioncited by 2
  10. 2025Explainable AI for Infection Prevention and Control: Modeling CPE Acquisition and Patient Outcomes in an Irish Hospital with Transformerscited by 1
  11. 2026Uncertainty-Aware Foundation Models for Clinical Datacited by 1
  12. 2025Trajectory-guided discharge stratification for heart failure using short-context electronic health record sequence modelingcited by 0
8 more
  1. 2026NEST: Nested Event Stream Transformer for Sequences of Multisetscited by 0
  2. 2026Dense Feature Learning via Linear Structure Preservation in Medical Datacited by 0
  3. 2026Event Fields: Learning Latent Event Structure for Waveform Foundation Modelscited by 0
  4. 2026WISTERIA: Learning Clinical Representations from Noisy Supervision via Multi-View Consistency in Electronic Health Recordscited by 0
  5. 2026EHR-RAGp: Retrieval-Augmented Prototype-Guided Foundation Model for Electronic Health Recordscited by 0
  6. 2026AURORA: Contextual Orthogonalization for Geometric Representation Learning in Healthcare Foundation Modelscited by 0
  7. 2026How Should Transformers Encode Numeric Values in Electronic Health Records?cited by 0
  8. 2026MorphologyFM: A Foundation Model for Morphology-Aware Representation Learning from ECG and Pulse Oximetry Waveformscited by 0