# Explainable Health Risk Predictor with Transformer-based Medicare Claim Encoder

arXiv: https://arxiv.org/abs/2105.09428 · HTML: https://mlsanity.com/paper/2105.09428/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 55 references that are in the map.
- First posted on arXiv 19 May 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [Scalable and accurate deep learning for electronic health records](https://mlsanity.com/paper/1801.07860/) (2018)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Doctor AI: Predicting Clinical Events via Recurrent Neural Networks](https://mlsanity.com/paper/1511.05942/) (2015)
- [GRAM: Graph-based Attention Model for Healthcare Representation Learning](https://mlsanity.com/paper/1611.07012/) (2016)
- [Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis](https://mlsanity.com/paper/1706.03446/) (2017)
- [Pre-training of Graph Augmented Transformers for Medication Recommendation](https://mlsanity.com/paper/1906.00346/) (2019)
- [Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks](https://mlsanity.com/paper/1706.05764/) (2017)
- [Language Models Are An Effective Patient Representation Learning Technique For Electronic Health Record Data](https://mlsanity.com/paper/2001.05295/) (2020)
- [Multi-layer Representation Learning for Medical Concepts](https://mlsanity.com/paper/1602.05568/) (2016)
- [RAIM: Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring Data](https://mlsanity.com/paper/1807.08820/) (2018)
- [Deep Representation Learning of Patient Data from Electronic Health Records (EHR): A Systematic Review](https://mlsanity.com/paper/2010.02809/) (2020)
- [MetaPred: Meta-Learning for Clinical Risk Prediction with Limited Patient Electronic Health Records](https://mlsanity.com/paper/1905.03218/) (2019)
- [Patient2Vec: A Personalized Interpretable Deep Representation of the Longitudinal Electronic Health Record](https://mlsanity.com/paper/1810.04793/) (2018)
- [Benchmarking Deep Learning Architectures for Predicting Readmission to the ICU and Describing Patients-at-Risk](https://mlsanity.com/paper/1905.08547/) (2019)
- [SECNLP: A Survey of Embeddings in Clinical Natural Language Processing](https://mlsanity.com/paper/1903.01039/) (2019)
- [How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection](https://mlsanity.com/paper/1902.10339/) (2019)
- [Improved Hierarchical Patient Classification with Language Model Pretraining over Clinical Notes](https://mlsanity.com/paper/1909.03039/) (2019)

## Built on by (most cited first, 4 of 4)

- [Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events](https://mlsanity.com/paper/2306.11547/) (2023)
- [Interpretable (not just posthoc-explainable) medical claims modeling for discharge placement to prevent avoidable all-cause readmissions or death](https://mlsanity.com/paper/2208.12814/) (2022)
- [Interpretable (not just posthoc-explainable) heterogeneous survivor bias-corrected treatment effects for assignment of postdischarge interventions to prevent readmissions](https://mlsanity.com/paper/2304.09981/) (2023)
- [Interpretable phenotyping of Heart Failure patients with Dutch discharge letters](https://mlsanity.com/paper/2505.24619/) (2025)

## Abstract (arXiv)

In 2019, The Centers for Medicare and Medicaid Services (CMS) launched an Artificial Intelligence (AI) Health Outcomes Challenge seeking solutions to predict risk in value-based care for incorporation into CMS Innovation Center payment and service delivery models. Recently, modern language models have played key roles in a number of health related tasks. This paper presents, to the best of our knowledge, the first application of these models to patient readmission prediction. To facilitate this, we create a dataset of 1.2 million medical history samples derived from the Limited Dataset (LDS) issued by CMS. Moreover, we propose a comprehensive modeling solution centered on a deep learning framework for this data. To demonstrate the framework, we train an attention-based Transformer to learn Medicare semantics in support of performing downstream prediction tasks thereby achieving 0.91 AUC and 0.91 recall on readmission classification. We also introduce a novel data pre-processing pipeline and discuss pertinent deployment considerations surrounding model explainability and bias.