# Sequential Diagnosis Prediction with Transformer and Ontological Representation

arXiv: https://arxiv.org/abs/2109.03069 · HTML: https://mlsanity.com/paper/2109.03069/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 7 Sep 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [GRAM: Graph-based Attention Model for Healthcare Representation Learning](https://mlsanity.com/paper/1611.07012/) (2016)
- [Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis](https://mlsanity.com/paper/1706.03446/) (2017)
- [Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks](https://mlsanity.com/paper/1706.05764/) (2017)
- [MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare](https://mlsanity.com/paper/1810.09593/) (2018)
- [Multi-layer Representation Learning for Medical Concepts](https://mlsanity.com/paper/1602.05568/) (2016)
- [Medical Concept Embedding with Time-Aware Attention](https://mlsanity.com/paper/1806.02873/) (2018)
- [Temporal Self-Attention Network for Medical Concept Embedding](https://mlsanity.com/paper/1909.06886/) (2019)
- [Federated Learning for Privacy-Preserving Open Innovation Future on Digital Health](https://mlsanity.com/paper/2108.10761/) (2021)
- [Self-Attention Enhanced Patient Journey Understanding in Healthcare System](https://mlsanity.com/paper/2006.10516/) (2020)
- [BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes](https://mlsanity.com/paper/2009.13252/) (2020)

## Built on by (most cited first, 12 of 12)

- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [Ontology Embedding: A Survey of Methods, Applications and Resources](https://mlsanity.com/paper/2406.10964/) (2024)
- [ICE-NODE: Integration of Clinical Embeddings with Neural Ordinary Differential Equations](https://mlsanity.com/paper/2207.01873/) (2022)
- [EMIT- Event-Based Masked Auto Encoding for Irregular Time Series](https://mlsanity.com/paper/2409.16554/) (2024)
- [Multi-Task Prediction of Clinical Outcomes in the Intensive Care Unit using Flexible Multimodal Transformers](https://mlsanity.com/paper/2111.05431/) (2021)
- [Deep Stable Representation Learning on Electronic Health Records](https://mlsanity.com/paper/2209.01321/) (2022)
- [Self-Explaining Hypergraph Neural Networks for Diagnosis Prediction](https://mlsanity.com/paper/2502.10689/) (2025)
- [Self Attention with Temporal Prior: Can We Learn More from Arrow of Time?](https://mlsanity.com/paper/2310.18932/) (2023)
- [Deep multimodal saliency parcellation of cerebellar pathways: linking microstructure and individual function through explainable multitask learning](https://mlsanity.com/paper/2407.15132/) (2024)
- [Detecci'on Autom'atica de Patolog'ias en Notas Cl'inicas en Espa~nol Combinando Modelos de Lenguaje y Ontolog'ias M'edicos](https://mlsanity.com/paper/2410.00616/) (2024)
- [Automatic detection of diseases in Spanish clinical notes combining medical language models and ontologies](https://mlsanity.com/paper/2412.03176/) (2024)
- [PRISM-Consult: A Panel-of-Experts Architecture for Clinician-Aligned Diagnosis](https://mlsanity.com/paper/2510.01114/) (2025)

## Abstract (arXiv)

Sequential diagnosis prediction on the Electronic Health Record (EHR) has been proven crucial for predictive analytics in the medical domain. EHR data, sequential records of a patient's interactions with healthcare systems, has numerous inherent characteristics of temporality, irregularity and data insufficiency. Some recent works train healthcare predictive models by making use of sequential information in EHR data, but they are vulnerable to irregular, temporal EHR data with the states of admission/discharge from hospital, and insufficient data. To mitigate this, we propose an end-to-end robust transformer-based model called SETOR, which exploits neural ordinary differential equation to handle both irregular intervals between a patient's visits with admitted timestamps and length of stay in each visit, to alleviate the limitation of insufficient data by integrating medical ontology, and to capture the dependencies between the patient's visits by employing multi-layer transformer blocks. Experiments conducted on two real-world healthcare datasets show that, our sequential diagnoses prediction model SETOR not only achieves better predictive results than previous state-of-the-art approaches, irrespective of sufficient or insufficient training data, but also derives more interpretable embeddings of medical codes. The experimental codes are available at the GitHub repository (https://github.com/Xueping/SETOR).