# Language Models Are An Effective Patient Representation Learning Technique For Electronic Health Record Data

arXiv: https://arxiv.org/abs/2001.05295 · HTML: https://mlsanity.com/paper/2001.05295/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jan 2020.

## Built on by (most cited first, 30 of 87)

- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models](https://mlsanity.com/paper/2307.02028/) (2023)
- [MOTOR: A Time-To-Event Foundation Model For Structured Medical Records](https://mlsanity.com/paper/2301.03150/) (2023)
- [Zero Shot Health Trajectory Prediction Using Transformer](https://mlsanity.com/paper/2407.21124/) (2024)
- [Multimodal LLMs for health grounded in individual-specific data](https://mlsanity.com/paper/2307.09018/) (2023)
- [Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events](https://mlsanity.com/paper/2306.11547/) (2023)
- [Benefits and Harms of Large Language Models in Digital Mental Health](https://mlsanity.com/paper/2311.14693/) (2023)
- [Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRs](https://mlsanity.com/paper/2412.16178/) (2024)
- [GenHPF: General Healthcare Predictive Framework with Multi-task Multi-source Learning](https://mlsanity.com/paper/2207.09858/) (2022)
- [CPLLM: Clinical Prediction with Large Language Models](https://mlsanity.com/paper/2309.11295/) (2023)
- [A Multi-Center Study on the Adaptability of a Shared Foundation Model for Electronic Health Records](https://mlsanity.com/paper/2311.11483/) (2023)
- [On the Challenges and Opportunities in Generative AI](https://mlsanity.com/paper/2403.00025/) (2024)
- [The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRs](https://mlsanity.com/paper/2303.12961/) (2023)
- [Large Language Models are Powerful Electronic Health Record Encoders](https://mlsanity.com/paper/2502.17403/) (2025)
- [Natural Language Processing in Electronic Health Records in Relation to Healthcare Decision-making: A Systematic Review](https://mlsanity.com/paper/2306.12834/) (2023)
- [Unifying Heterogeneous Electronic Health Records Systems via Text-Based Code Embedding](https://mlsanity.com/paper/2108.03625/) (2021)
- [Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicine](https://mlsanity.com/paper/2311.04937/) (2023)
- [Emergency Department Decision Support using Clinical Pseudo-notes](https://mlsanity.com/paper/2402.00160/) (2024)
- [Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?](https://mlsanity.com/paper/2301.09017/) (2023)
- [When Geoscience Meets Foundation Models: Towards General Geoscience Artificial Intelligence System](https://mlsanity.com/paper/2309.06799/) (2023)
- [Event-Based Contrastive Learning for Medical Time Series](https://mlsanity.com/paper/2312.10308/) (2023)
- [RelayAttention for Efficient Large Language Model Serving with Long System Prompts](https://mlsanity.com/paper/2402.14808/) (2024)
- [A comparison of approaches to improve worst-case predictive model performance over patient subpopulations](https://mlsanity.com/paper/2108.12250/) (2021)
- [General-Purpose Retrieval-Enhanced Medical Prediction Model Using Near-Infinite History](https://mlsanity.com/paper/2310.20204/) (2023)
- [ChronoFormer: Time-Aware Transformer Architectures for Structured Clinical Event Modeling](https://mlsanity.com/paper/2504.07373/) (2025)
- [Foundation models for electronic health records: representation dynamics and transferability](https://mlsanity.com/paper/2504.10422/) (2025)
- [Leveraging Time Irreversibility with Order-Contrastive Pre-training](https://mlsanity.com/paper/2111.02599/) (2021)

## Abstract (arXiv)

Widespread adoption of electronic health records (EHRs) has fueled the development of using machine learning to build prediction models for various clinical outcomes. This process is often constrained by having a relatively small number of patient records for training the model. We demonstrate that using patient representation schemes inspired from techniques in natural language processing can increase the accuracy of clinical prediction models by transferring information learned from the entire patient population to the task of training a specific model, where only a subset of the population is relevant. Such patient representation schemes enable a 3.5% mean improvement in AUROC on five prediction tasks compared to standard baselines, with the average improvement rising to 19% when only a small number of patient records are available for training the clinical prediction model.