# Time-Aware Attention for Enhanced Electronic Health Records Modeling

arXiv: https://arxiv.org/abs/2507.14847 · HTML: https://mlsanity.com/paper/2507.14847/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 20 Jul 2025.

## Builds on (most cited first, 16 of 16)

- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks](https://mlsanity.com/paper/1706.05764/) (2017)
- [MOTOR: A Time-To-Event Foundation Model For Structured Medical Records](https://mlsanity.com/paper/2301.03150/) (2023)
- [CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks](https://mlsanity.com/paper/2111.08585/) (2021)
- [Unifying Heterogeneous Electronic Health Records Systems via Text-Based Code Embedding](https://mlsanity.com/paper/2108.03625/) (2021)
- [MediTab: Scaling Medical Tabular Data Predictors via Data Consolidation, Enrichment, and Refinement](https://mlsanity.com/paper/2305.12081/) (2023)
- [Emergency Department Decision Support using Clinical Pseudo-notes](https://mlsanity.com/paper/2402.00160/) (2024)
- [Multimodal Medical Code Tokenizer](https://mlsanity.com/paper/2502.04397/) (2025)
- [Event-Based Contrastive Learning for Medical Time Series](https://mlsanity.com/paper/2312.10308/) (2023)
- [General-Purpose Retrieval-Enhanced Medical Prediction Model Using Near-Infinite History](https://mlsanity.com/paper/2310.20204/) (2023)
- [Learning the Joint Representation of Heterogeneous Temporal Events for Clinical Endpoint Prediction](https://mlsanity.com/paper/1803.04837/) (2018)
- [Modeling Inter-Dependence Between Time and Mark in Multivariate Temporal Point Processes](https://mlsanity.com/paper/2210.15294/) (2022)
- [Interpretable Neural Temporal Point Processes for Modelling Electronic Health Records](https://mlsanity.com/paper/2404.08007/) (2024)

## Built on by (most cited first, 3 of 3)

- [Latent Factor Point Processes for Patient Representation in Electronic Health Records](https://mlsanity.com/paper/2508.20327/) (2025)
- [MATA-Former & SIICU: Semantic Aware Temporal Alignment for High-Fidelity ICU Risk Prediction](https://mlsanity.com/paper/2604.01727/) (2026)
- [NOAH: Learning the Full Patient Journey. A Longitudinal Multimodal Time-Aware Model for Representation and Forecasting](https://mlsanity.com/paper/2609.09140/) (2026)

## Abstract (arXiv)

Electronic Health Records (EHR) contain valuable clinical information for predicting patient outcomes and guiding healthcare decisions. However, effectively modeling Electronic Health Records (EHRs) requires addressing data heterogeneity and complex temporal patterns. Standard approaches often struggle with irregular time intervals between clinical events. We propose TALE-EHR, a Transformer-based framework featuring a novel time-aware attention mechanism that explicitly models continuous temporal gaps to capture fine-grained sequence dynamics. To complement this temporal modeling with robust semantics, TALE-EHR leverages embeddings derived from standardized code descriptions using a pre-trained Large Language Model (LLM), providing a strong foundation for understanding clinical concepts. Experiments on the MIMIC-IV and PIC dataset demonstrate that our approach outperforms state-of-the-art baselines on tasks such as disease progression forecasting. TALE-EHR underscores the benefit of integrating explicit, continuous temporal modeling with strong semantic representations provides a powerful solution for advancing EHR analysis.