# Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologies

arXiv: https://arxiv.org/abs/2107.09951 · HTML: https://mlsanity.com/paper/2107.09951/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 167 references that are in the map.
- First posted on arXiv 21 Jul 2021.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [Scalable and accurate deep learning for electronic health records](https://mlsanity.com/paper/1801.07860/) (2018)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Doctor AI: Predicting Clinical Events via Recurrent Neural Networks](https://mlsanity.com/paper/1511.05942/) (2015)
- [Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions](https://mlsanity.com/paper/1909.00590/) (2019)
- [Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis](https://mlsanity.com/paper/1706.03446/) (2017)
- [The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations](https://mlsanity.com/paper/1907.09294/) (2019)
- [Supersparse Linear Integer Models for Optimized Medical Scoring Systems](https://mlsanity.com/paper/1502.04269/) (2015)
- [MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare](https://mlsanity.com/paper/1810.09593/) (2018)
- [Multi-layer Representation Learning for Medical Concepts](https://mlsanity.com/paper/1602.05568/) (2016)
- [RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records](https://mlsanity.com/paper/1805.10724/) (2018)
- [Deepr: A Convolutional Net for Medical Records](https://mlsanity.com/paper/1607.07519/) (2016)
- [StageNet: Stage-Aware Neural Networks for Health Risk Prediction](https://mlsanity.com/paper/2001.10054/) (2020)
- [Learning Optimized Risk Scores](https://mlsanity.com/paper/1610.00168/) (2016)
- [Deep Representation Learning of Patient Data from Electronic Health Records (EHR): A Systematic Review](https://mlsanity.com/paper/2010.02809/) (2020)
- [Deep Representation Learning of Electronic Health Records to Unlock Patient Stratification at Scale](https://mlsanity.com/paper/2003.06516/) (2020)
- [COMPOSE: Cross-Modal Pseudo-Siamese Network for Patient Trial Matching](https://mlsanity.com/paper/2006.08765/) (2020)
- [Boosting Deep Learning Risk Prediction with Generative Adversarial Networks for Electronic Health Records](https://mlsanity.com/paper/1709.01648/) (2017)

## Built on by (most cited first, 30 of 34)

- [Benchmarking emergency department triage prediction models with machine learning and large public electronic health records](https://mlsanity.com/paper/2111.11017/) (2021)
- [A scoping review on multimodal deep learning in biomedical images and texts](https://mlsanity.com/paper/2307.07362/) (2023)
- [CTPD: Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis](https://mlsanity.com/paper/2411.00696/) (2024)
- [An empirical study of the effect of background data size on the stability of SHapley Additive exPlanations (SHAP) for deep learning models](https://mlsanity.com/paper/2204.11351/) (2022)
- [Multi-View Contrastive Learning for Robust Domain Adaptation in Medical Time Series Analysis](https://mlsanity.com/paper/2506.22393/) (2025)
- [TrajSurv: Learning Continuous Latent Trajectories from Electronic Health Records for Trustworthy Survival Prediction](https://mlsanity.com/paper/2508.00657/) (2025)
- [Cross-Representation Benchmarking in Time-Series Electronic Health Records for Clinical Outcome Prediction](https://mlsanity.com/paper/2510.09159/) (2025)
- [Leveraging Anatomical Constraints with Uncertainty for Pneumothorax Segmentation](https://mlsanity.com/paper/2311.15213/) (2023)
- [Advancing Precision Oncology Through Modeling of Longitudinal and Multimodal Data](https://mlsanity.com/paper/2502.07836/) (2025)
- [Organ-Agents: Virtual Human Physiology Simulator via LLMs](https://mlsanity.com/paper/2508.14357/) (2025)
- [The Taxonomies, Training, and Applications of Event Stream Modelling for Electronic Health Records](https://mlsanity.com/paper/2603.14003/) (2026)
- [Embedding-based Multimodal Learning on Pan-Squamous Cell Carcinomas for Improved Survival Outcomes](https://mlsanity.com/paper/2406.08521/) (2024)
- [Bi-Axial Transformers: Addressing the Increasing Complexity of EHR Classification](https://mlsanity.com/paper/2508.12418/) (2025)
- [Shapley variable importance cloud for machine learning models](https://mlsanity.com/paper/2212.08370/) (2022)
- [Clinical Domain Knowledge-Derived Template Improves Post Hoc AI Explanations in Pneumothorax Classification](https://mlsanity.com/paper/2403.18871/) (2024)
- [Explainable Artificial Intelligence Techniques for Irregular Temporal Classification of Multidrug Resistance Acquisition in Intensive Care Unit Patients](https://mlsanity.com/paper/2407.17165/) (2024)
- [Explainable Spatio-Temporal GCNNs for Irregular Multivariate Time Series: Architecture and Application to ICU Patient Data](https://mlsanity.com/paper/2411.01070/) (2024)
- [PA-RNet: Perturbation-Aware Residual Network for Robust Multimodal Time Series Forecasting](https://mlsanity.com/paper/2508.04750/) (2025)
- [User-Adaptive Meta-Learning for Cold-Start Medication Recommendation with Uncertainty Filtering](https://mlsanity.com/paper/2601.22820/) (2026)
- [Capture Timing-Attention of Events in Clinical Time Series](https://mlsanity.com/paper/2602.10385/) (2026)
- [DoctorAgents: an agentic framework to iteratively refine AutoML pipeline for small clinical temporal data](https://mlsanity.com/paper/2608.05375/) (2026)
- [tSPM+; a high-performance algorithm for mining transitive sequential patterns from clinical data](https://mlsanity.com/paper/2309.05671/) (2023)
- [Foundation Model Makes Clustering A Better Initialization For Cold-Start Active Learning](https://mlsanity.com/paper/2402.02561/) (2024)
- [Explainable AI for Classifying UTI Risk Groups Using a Real-World Linked EHR and Pathology Lab Dataset](https://mlsanity.com/paper/2411.17645/) (2024)
- [PT: A Plain Transformer is Good Hospital Readmission Predictor](https://mlsanity.com/paper/2412.12909/) (2024)
- [Early Detection of Multidrug Resistance Using Multivariate Time Series Analysis and Interpretable Patient-Similarity Representations](https://mlsanity.com/paper/2504.17717/) (2025)
- [MedM2T: A MultiModal Framework for Time-Aware Modeling with Electronic Health Record and Electrocardiogram Data](https://mlsanity.com/paper/2510.27321/) (2025)
- [BiPETE: A Bi-Positional Embedding Transformer Encoder for Risk Assessment of Alcohol and Substance Use Disorder with Electronic Health Records](https://mlsanity.com/paper/2511.04998/) (2025)
- [MAPI-GNN: Multi-Activation Plane Interaction Graph Neural Network for Multimodal Medical Diagnosis](https://mlsanity.com/paper/2512.20026/) (2025)
- [Temporal Fusion Nexus: A task-agnostic multi-modal embedding model for clinical narratives and irregular time series in post-kidney transplant care](https://mlsanity.com/paper/2601.08503/) (2026)

## Abstract (arXiv)

Objective: Temporal electronic health records (EHRs) can be a wealth of information for secondary uses, such as clinical events prediction or chronic disease management. However, challenges exist for temporal data representation. We therefore sought to identify these challenges and evaluate novel methodologies for addressing them through a systematic examination of deep learning solutions. Methods: We searched five databases (PubMed, EMBASE, the Institute of Electrical and Electronics Engineers [IEEE] Xplore Digital Library, the Association for Computing Machinery [ACM] digital library, and Web of Science) complemented with hand-searching in several prestigious computer science conference proceedings. We sought articles that reported deep learning methodologies on temporal data representation in structured EHR data from January 1, 2010, to August 30, 2020. We summarized and analyzed the selected articles from three perspectives: nature of time series, methodology, and model implementation. Results: We included 98 articles related to temporal data representation using deep learning. Four major challenges were identified, including data irregularity, data heterogeneity, data sparsity, and model opacity. We then studied how deep learning techniques were applied to address these challenges. Finally, we discuss some open challenges arising from deep learning. Conclusion: Temporal EHR data present several major challenges for clinical prediction modeling and data utilization. To some extent, current deep learning solutions can address these challenges. Future studies can consider designing comprehensive and integrated solutions. Moreover, researchers should incorporate additional clinical domain knowledge into study designs and enhance the interpretability of the model to facilitate its implementation in clinical practice.