# DuETT: Dual Event Time Transformer for Electronic Health Records

arXiv: https://arxiv.org/abs/2304.13017 · HTML: https://mlsanity.com/paper/2304.13017/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [TSMixer: An All-MLP Architecture for Time Series Forecasting](https://mlsanity.com/paper/2303.06053/) (2023)
- [Multi-Time Attention Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/2101.10318/) (2021)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [Transformers without Tears: Improving the Normalization of Self-Attention](https://mlsanity.com/paper/1910.05895/) (2019)
- [Graph-Guided Network for Irregularly Sampled Multivariate Time Series](https://mlsanity.com/paper/2110.05357/) (2021)
- [Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series](https://mlsanity.com/paper/2107.14293/) (2021)
- [Learning Long-Term Dependencies in Irregularly-Sampled Time Series](https://mlsanity.com/paper/2006.04418/) (2020)
- [Hydra Attention: Efficient Attention with Many Heads](https://mlsanity.com/paper/2209.07484/) (2022)

## Built on by (most cited first, 17 of 17)

- [Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions](https://mlsanity.com/paper/2507.00191/) (2025)
- [Event-Based Contrastive Learning for Medical Time Series](https://mlsanity.com/paper/2312.10308/) (2023)
- [MEDS-Tab: Automated tabularization and baseline methods for MEDS datasets](https://mlsanity.com/paper/2411.00200/) (2024)
- [Towards Practical Multi-label Causal Discovery in High-Dimensional Event Sequences via One-Shot Graph Aggregation](https://mlsanity.com/paper/2509.19112/) (2025)
- [One-Shot Multi-Label Causal Discovery in High-Dimensional Event Sequences](https://mlsanity.com/paper/2509.23213/) (2025)
- [Towards Unified Approaches in Self-Supervised Event Stream Modeling: Progress and Prospects](https://mlsanity.com/paper/2502.04899/) (2025)
- [SuperMAN: Interpretable and Expressive Networks over Temporally Sparse Heterogeneous Data](https://mlsanity.com/paper/2505.19193/) (2025)
- [Towards Self-Supervised Foundation Models for Critical Care Time Series](https://mlsanity.com/paper/2509.19885/) (2025)
- [PULSE-ICU: A Pretrained Unified Long-Sequence Encoder for Multi-task Prediction in Intensive Care Units](https://mlsanity.com/paper/2511.22199/) (2025)
- [Transforming Vehicle Diagnostics: A Multimodal Approach to Error Patterns Prediction](https://mlsanity.com/paper/2602.01109/) (2026)
- [Multi-Agent Causal Reasoning System for Error Pattern Rule Automation in Vehicles](https://mlsanity.com/paper/2602.01155/) (2026)
- [Learning Representations from Incomplete EHR Data with Dual-Masked Autoencoding](https://mlsanity.com/paper/2602.15159/) (2026)
- [Structure-Aware Set Transformers: Temporal and Variable-Type Attention Biases for Asynchronous Clinical Time Series](https://mlsanity.com/paper/2603.06605/) (2026)
- [DBGL: Decay-aware Bipartite Graph Learning for Irregular Medical Time Series Classification](https://mlsanity.com/paper/2604.11842/) (2026)
- [Representation Before Training: A Practical Benchmark for Generative Medical Event Model Tokenization](https://mlsanity.com/paper/2604.16775/) (2026)
- [Learning to Select Source-Traceable Evidence for Language-Model Prediction from Irregular Clinical Time Series](https://mlsanity.com/paper/2605.20292/) (2026)
- [Rethinking How We Evaluate Methodological Progress in Health AI](https://mlsanity.com/paper/2609.18134/) (2026)

## Abstract (arXiv)

Electronic health records (EHRs) recorded in hospital settings typically contain a wide range of numeric time series data that is characterized by high sparsity and irregular observations. Effective modelling for such data must exploit its time series nature, the semantic relationship between different types of observations, and information in the sparsity structure of the data. Self-supervised Transformers have shown outstanding performance in a variety of structured tasks in NLP and computer vision. But multivariate time series data contains structured relationships over two dimensions: time and recorded event type, and straightforward applications of Transformers to time series data do not leverage this distinct structure. The quadratic scaling of self-attention layers can also significantly limit the input sequence length without appropriate input engineering. We introduce the DuETT architecture, an extension of Transformers designed to attend over both time and event type dimensions, yielding robust representations from EHR data. DuETT uses an aggregated input where sparse time series are transformed into a regular sequence with fixed length; this lowers the computational complexity relative to previous EHR Transformer models and, more importantly, enables the use of larger and deeper neural networks. When trained with self-supervised prediction tasks, that provide rich and informative signals for model pre-training, our model outperforms state-of-the-art deep learning models on multiple downstream tasks from the MIMIC-IV and PhysioNet-2012 EHR datasets.