# Transformer-based unsupervised patient representation learning based on medical claims for risk stratification and analysis

arXiv: https://arxiv.org/abs/2106.12658 · HTML: https://mlsanity.com/paper/2106.12658/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 33 references that are in the map.
- First posted on arXiv 23 Jun 2021.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Doctor AI: Predicting Clinical Events via Recurrent Neural Networks](https://mlsanity.com/paper/1511.05942/) (2015)
- [GRAM: Graph-based Attention Model for Healthcare Representation Learning](https://mlsanity.com/paper/1611.07012/) (2016)
- [Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks](https://mlsanity.com/paper/1706.05764/) (2017)
- [Multi-layer Representation Learning for Medical Concepts](https://mlsanity.com/paper/1602.05568/) (2016)
- [Deep Representation Learning of Electronic Health Records to Unlock Patient Stratification at Scale](https://mlsanity.com/paper/2003.06516/) (2020)
- [General-Purpose User Embeddings based on Mobile App Usage](https://mlsanity.com/paper/2005.13303/) (2020)
- [Healthcare Cost Prediction: Leveraging Fine-grain Temporal Patterns](https://mlsanity.com/paper/2009.06780/) (2020)

## Built on by (most cited first, 4 of 4)

- [Similar Data Points Identification with LLM: A Human-in-the-loop Strategy Using Summarization and Hidden State Insights](https://mlsanity.com/paper/2404.04281/) (2024)
- [Privacy-preserving patient clustering for personalized federated learning](https://mlsanity.com/paper/2307.08847/) (2023)
- [Translating Expert Intuition into Quantifiable Features: Encode Investigator Domain Knowledge via LLM for Enhanced Predictive Analytics](https://mlsanity.com/paper/2405.08017/) (2024)
- [A Customer Level Fraudulent Activity Detection Benchmark for Enhancing Machine Learning Model Research and Evaluation](https://mlsanity.com/paper/2404.14746/) (2024)

## Abstract (arXiv)

The claims data, containing medical codes, services information, and incurred expenditure, can be a good resource for estimating an individual's health condition and medical risk level. In this study, we developed Transformer-based Multimodal AutoEncoder (TMAE), an unsupervised learning framework that can learn efficient patient representation by encoding meaningful information from the claims data. TMAE is motivated by the practical needs in healthcare to stratify patients into different risk levels for improving care delivery and management. Compared to previous approaches, TMAE is able to 1) model inpatient, outpatient, and medication claims collectively, 2) handle irregular time intervals between medical events, 3) alleviate the sparsity issue of the rare medical codes, and 4) incorporate medical expenditure information. We trained TMAE using a real-world pediatric claims dataset containing more than 600,000 patients and compared its performance with various approaches in two clustering tasks. Experimental results demonstrate that TMAE has superior performance compared to all baselines. Multiple downstream applications are also conducted to illustrate the effectiveness of our framework. The promising results confirm that the TMAE framework is scalable to large claims data and is able to generate efficient patient embeddings for risk stratification and analysis.