# Targeted-BEHRT: Deep learning for observational causal inference on longitudinal electronic health records

arXiv: https://arxiv.org/abs/2202.03487 · HTML: https://mlsanity.com/paper/2202.03487/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 61 references that are in the map.
- First posted on arXiv 7 Feb 2022.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Adapting Neural Networks for the Estimation of Treatment Effects](https://mlsanity.com/paper/1906.02120/) (2019)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Bayesian Inference of Individualized Treatment Effects using Multi-task Gaussian Processes](https://mlsanity.com/paper/1704.02801/) (2017)
- [Auxiliary Tasks in Multi-task Learning](https://mlsanity.com/paper/1805.06334/) (2018)
- [Generalization Bounds and Representation Learning for Estimation of Potential Outcomes and Causal Effects](https://mlsanity.com/paper/2001.07426/) (2020)
- [RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records](https://mlsanity.com/paper/1805.10724/) (2018)
- [Deepr: A Convolutional Net for Medical Records](https://mlsanity.com/paper/1607.07519/) (2016)
- [An explainable Transformer-based deep learning model for the prediction of incident heart failure](https://mlsanity.com/paper/2101.11359/) (2021)
- [Adapting Text Embeddings for Causal Inference](https://mlsanity.com/paper/1905.12741/) (2019)
- [Targeted Learning with Daily EHR Data](https://mlsanity.com/paper/1705.09874/) (2017)
- [An Easy Implementation of CV-TMLE](https://mlsanity.com/paper/1811.04573/) (2018)

## Built on by (most cited first, 9 of 9)

- [Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events](https://mlsanity.com/paper/2306.11547/) (2023)
- [CORE-BEHRT: A Carefully Optimized and Rigorously Evaluated BEHRT](https://mlsanity.com/paper/2404.15201/) (2024)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [ExBEHRT: Extended Transformer for Electronic Health Records to Predict Disease Subtypes & Progressions](https://mlsanity.com/paper/2303.12364/) (2023)
- [A Survey of Deep Causal Models and Their Industrial Applications](https://mlsanity.com/paper/2209.08860/) (2022)
- [CEHR-XGPT: A Scalable Multi-Task Foundation Model for Electronic Health Records](https://mlsanity.com/paper/2509.03643/) (2025)
- [MuST: Multimodal Spatiotemporal Graph-Transformer for Hospital Readmission Prediction](https://mlsanity.com/paper/2311.07608/) (2023)
- [Advancing Precision Oncology Through Modeling of Longitudinal and Multimodal Data](https://mlsanity.com/paper/2502.07836/) (2025)
- [Trajectory-guided discharge stratification for heart failure using short-context electronic health record sequence modeling](https://mlsanity.com/paper/2511.16839/) (2025)

## Abstract (arXiv)

Observational causal inference is useful for decision making in medicine when randomized clinical trials (RCT) are infeasible or non generalizable. However, traditional approaches fail to deliver unconfounded causal conclusions in practice. The rise of "doubly robust" non-parametric tools coupled with the growth of deep learning for capturing rich representations of multimodal data, offers a unique opportunity to develop and test such models for causal inference on comprehensive electronic health records (EHR). In this paper, we investigate causal modelling of an RCT-established null causal association: the effect of antihypertensive use on incident cancer risk. We develop a dataset for our observational study and a Transformer-based model, Targeted BEHRT coupled with doubly robust estimation, we estimate average risk ratio (RR). We compare our model to benchmark statistical and deep learning models for causal inference in multiple experiments on semi-synthetic derivations of our dataset with various types and intensities of confounding. In order to further test the reliability of our approach, we test our model on situations of limited data. We find that our model provides more accurate estimates of RR (least sum absolute error from ground truth) compared to benchmarks for risk ratio estimation on high-dimensional EHR across experiments. Finally, we apply our model to investigate the original case study: antihypertensives' effect on cancer and demonstrate that our model generally captures the validated null association.