# TEE4EHR: Transformer Event Encoder for Better Representation Learning in Electronic Health Records

arXiv: https://arxiv.org/abs/2402.06367 · HTML: https://mlsanity.com/paper/2402.06367/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 56 references that are in the map.
- First posted on arXiv 9 Feb 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [Multi-Time Attention Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/2101.10318/) (2021)
- [Transformer Hawkes Process](https://mlsanity.com/paper/2002.09291/) (2020)
- [Set Functions for Time Series](https://mlsanity.com/paper/1909.12064/) (2019)
- [Neural Temporal Point Processes: A Review](https://mlsanity.com/paper/2104.03528/) (2021)
- [Graph-Guided Network for Irregularly Sampled Multivariate Time Series](https://mlsanity.com/paper/2110.05357/) (2021)
- [Interpolation-Prediction Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/1909.07782/) (2019)
- [Deep Reinforcement Learning for Sepsis Treatment](https://mlsanity.com/paper/1711.09602/) (2017)
- [Learning Granger Causality for Hawkes Processes](https://mlsanity.com/paper/1602.04511/) (2016)
- [A Review of Challenges and Opportunities in Machine Learning for Health](https://mlsanity.com/paper/1806.00388/) (2018)
- [What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding](https://mlsanity.com/paper/2010.04903/) (2020)
- [Neural Temporal Point Processes For Modelling Electronic Health Records](https://mlsanity.com/paper/2007.13794/) (2020)
- [Deep Reinforcement Learning of Marked Temporal Point Processes](https://mlsanity.com/paper/1805.09360/) (2018)
- [Learning from Clinical Judgments: Semi-Markov-Modulated Marked Hawkes Processes for Risk Prognosis](https://mlsanity.com/paper/1705.05267/) (2017)
- [Attention based Convolutional Recurrent Neural Network for Environmental Sound Classification](https://mlsanity.com/paper/1907.02230/) (2019)
- [Time-Parameterized Convolutional Neural Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/2308.03210/) (2023)

## Built on by (most cited first, 3 of 3)

- [The Taxonomies, Training, and Applications of Event Stream Modelling for Electronic Health Records](https://mlsanity.com/paper/2603.14003/) (2026)
- [Patient foundation model for risk stratification in low-risk overweight patients](https://mlsanity.com/paper/2602.09079/) (2026)
- [Deep Kernel Learning for Stratifying Glaucoma Trajectories](https://mlsanity.com/paper/2605.00708/) (2026)

## Abstract (arXiv)

Irregular sampling of time series in electronic health records (EHRs) is one of the main challenges for developing machine learning models. Additionally, the pattern of missing data in certain clinical variables is not at random but depends on the decisions of clinicians and the state of the patient. Point process is a mathematical framework for analyzing event sequence data that is consistent with irregular sampling patterns. Our model, TEE4EHR, is a transformer event encoder (TEE) with point process loss that encodes the pattern of laboratory tests in EHRs. The utility of our TEE has been investigated in a variety of benchmark event sequence datasets. Additionally, we conduct experiments on two real-world EHR databases to provide a more comprehensive evaluation of our model. Firstly, in a self-supervised learning approach, the TEE is jointly learned with an existing attention-based deep neural network which gives superior performance in negative log-likelihood and future event prediction. Besides, we propose an algorithm for aggregating attention weights that can reveal the interaction between the events. Secondly, we transfer and freeze the learned TEE to the downstream task for the outcome prediction, where it outperforms state-of-the-art models for handling irregularly sampled time series. Furthermore, our results demonstrate that our approach can improve representation learning in EHRs and can be useful for clinical prediction tasks.