# Medical Profile Model: Scientific and Practical Applications in Healthcare

arXiv: https://arxiv.org/abs/2107.03913 · HTML: https://mlsanity.com/paper/2107.03913/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 40 references that are in the map.
- First posted on arXiv 21 Jun 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [A Comprehensive Survey of Graph Embedding: Problems, Techniques and Applications](https://mlsanity.com/paper/1709.07604/) (2017)
- [Doctor AI: Predicting Clinical Events via Recurrent Neural Networks](https://mlsanity.com/paper/1511.05942/) (2015)
- [Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks](https://mlsanity.com/paper/1706.05764/) (2017)
- [Attend and Diagnose: Clinical Time Series Analysis using Attention Models](https://mlsanity.com/paper/1711.03905/) (2017)
- [DeepCare: A Deep Dynamic Memory Model for Predictive Medicine](https://mlsanity.com/paper/1602.00357/) (2016)
- [Self-Attention Enhanced Patient Journey Understanding in Healthcare System](https://mlsanity.com/paper/2006.10516/) (2020)
- [Predicting Clinical Diagnosis from Patients Electronic Health Records Using BERT-based Neural Networks](https://mlsanity.com/paper/2007.07562/) (2020)
- [BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes](https://mlsanity.com/paper/2009.13252/) (2020)

## Built on by (most cited first, 2 of 2)

- [Efficient Generative Prediction for EHR Foundation Models: The SCOPE and REACH Estimators](https://mlsanity.com/paper/2602.03730/) (2026)
- [Can-SAVE: Deploying Low-Cost and Population-Scale Cancer Screening via Survival Analysis Variables and EHR](https://mlsanity.com/paper/2309.15039/) (2023)

## Abstract (arXiv)

The paper researches the problem of representation learning for electronic health records. We present the patient histories as temporal sequences of diseases for which embeddings are learned in an unsupervised setup with a transformer-based neural network model. Additionally the embedding space includes demographic parameters which allow the creation of generalized patient profiles and successful transfer of medical knowledge to other domains. The training of such a medical profile model has been performed on a dataset of more than one million patients. Detailed model analysis and its comparison with the state-of-the-art method show its clear advantage in the diagnosis prediction task. Further, we show two applications based on the developed profile model. First, a novel Harbinger Disease Discovery method allowing to reveal disease associated hypotheses and potentially are beneficial in the design of epidemiological studies. Second, the patient embeddings extracted from the profile model applied to the insurance scoring task allow significant improvement in the performance metrics.