# Large Language Models are Powerful Electronic Health Record Encoders

arXiv: https://arxiv.org/abs/2502.17403 · HTML: https://mlsanity.com/paper/2502.17403/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 80 references that are in the map.
- First posted on arXiv 24 Feb 2025.

## Builds on (most cited first, 30 of 36)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Scalable and accurate deep learning for electronic health records](https://mlsanity.com/paper/1801.07860/) (2018)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://mlsanity.com/paper/2107.07653/) (2021)
- [Generative Representational Instruction Tuning](https://mlsanity.com/paper/2402.09906/) (2024)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study](https://mlsanity.com/paper/2305.13062/) (2023)
- [EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models](https://mlsanity.com/paper/2307.02028/) (2023)
- [LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding](https://mlsanity.com/paper/2104.08836/) (2021)
- [Language Models Are An Effective Patient Representation Learning Technique For Electronic Health Record Data](https://mlsanity.com/paper/2001.05295/) (2020)
- [CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks](https://mlsanity.com/paper/2111.08585/) (2021)
- [Repetition Improves Language Model Embeddings](https://mlsanity.com/paper/2402.15449/) (2024)
- [CPLLM: Clinical Prediction with Large Language Models](https://mlsanity.com/paper/2309.11295/) (2023)
- [ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?](https://mlsanity.com/paper/2411.06469/) (2024)
- [Prompting Large Language Models for Zero-Shot Clinical Prediction with Structured Longitudinal Electronic Health Record Data](https://mlsanity.com/paper/2402.01713/) (2024)
- [CORE-BEHRT: A Carefully Optimized and Rigorously Evaluated BEHRT](https://mlsanity.com/paper/2404.15201/) (2024)

## Built on by (most cited first, 26 of 26)

- [EHR-R1: A Reasoning-Enhanced Foundational Language Model for Electronic Health Record Analysis](https://mlsanity.com/paper/2510.25628/) (2025)
- [ChronoFormer: Time-Aware Transformer Architectures for Structured Clinical Event Modeling](https://mlsanity.com/paper/2504.07373/) (2025)
- [EHR-RAG: Bridging Long-Horizon Structured Electronic Health Records and Large Language Models via Enhanced Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.21340/) (2026)
- [Fusion or Confusion? Multimodal Complexity Is Not All You Need](https://mlsanity.com/paper/2512.22991/) (2025)
- [One Loss to Rule Them All: Marked Time-to-Event for Structured EHR Foundation Models](https://mlsanity.com/paper/2602.00541/) (2026)
- [AgentEHR: Advancing Autonomous Clinical Decision-Making via Retrospective Summarization](https://mlsanity.com/paper/2601.13918/) (2026)
- [Rethinking Tokenization for Clinical Time Series: When Less is More](https://mlsanity.com/paper/2512.05217/) (2025)
- [LLMs can construct powerful representations and streamline sample-efficient supervised learning](https://mlsanity.com/paper/2603.11679/) (2026)
- [Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning](https://mlsanity.com/paper/2604.05834/) (2026)
- [PORTER: Language-Grounded Event Representations for Portable Structured EHR Foundation Models](https://mlsanity.com/paper/2606.24102/) (2026)
- [The CRITICAL Records Integrated Standardization Pipeline (CRISP): End-to-End Processing of Large-scale Multi-institutional OMOP CDM Data](https://mlsanity.com/paper/2509.08247/) (2025)
- [PaReGTA: A Temporally Aware LLM-Based Patient Representation Framework for EHR Analytics](https://mlsanity.com/paper/2602.19661/) (2026)
- [Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning](https://mlsanity.com/paper/2607.17673/) (2026)
- [APRIL: Annotations for Policy evaluation with Reliable Inference from LLMs](https://mlsanity.com/paper/2511.17818/) (2025)
- [Aligning Probabilistic Beliefs under Informative Missingness: LLM Steerability in Clinical Reasoning](https://mlsanity.com/paper/2512.00479/) (2025)
- [CacheMind: From Miss Rates to Why -- Natural-Language, Trace-Grounded Reasoning for Cache Replacement](https://mlsanity.com/paper/2602.12422/) (2026)
- [MATA-Former & SIICU: Semantic Aware Temporal Alignment for High-Fidelity ICU Risk Prediction](https://mlsanity.com/paper/2604.01727/) (2026)
- [Representation Before Training: A Practical Benchmark for Generative Medical Event Model Tokenization](https://mlsanity.com/paper/2604.16775/) (2026)
- [OphthaDT: Generative Digital Twins for Forecasting Visual Acuity Trajectories in Ophthalmology](https://mlsanity.com/paper/2606.22101/) (2026)
- [A General Multimodal Probing Framework for Data Fusion and Prediction with Frozen Large Language Models: Evidence from Multimodal EHR Data](https://mlsanity.com/paper/2606.28798/) (2026)
- [Large Language Models as Unified Multimodal Learners for Clinical Prediction](https://mlsanity.com/paper/2607.15380/) (2026)
- [Patients-like-me: A Variational LM--GNN Framework for Explainable Clinical Prediction](https://mlsanity.com/paper/2608.04193/) (2026)
- [Foresight-England: Development of a National-Scale Generative AI Model of Electronic Health Records for Medical Event Prediction across the COVID-19 Pandemic](https://mlsanity.com/paper/2608.16273/) (2026)
- [EHRAdapt: Adapting Pretrained Language Models to Electronic Health Records with Semantic Priors for Rare Clinical Events](https://mlsanity.com/paper/2609.34007/) (2026)
- [GLoC-EHR: Evidence-Cited Clinical Reasoning over Global Context and Local EHR Events](https://mlsanity.com/paper/2610.01076/) (2026)
- [Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning](https://mlsanity.com/paper/2610.14770/) (2026)

## Abstract (arXiv)

Electronic Health Records (EHRs) offer considerable potential for clinical prediction, but their complexity and heterogeneity challenge traditional machine learning. Domain-specific EHR foundation models trained on unlabeled EHR data have shown improved predictive accuracy and generalization. However, their development is constrained by limited data access and site-specific vocabularies. We convert EHR data into plain text by replacing medical codes with natural-language descriptions, enabling general-purpose Large Language Models (LLMs) to produce high-dimensional embeddings for downstream prediction tasks without access to private medical training data. LLM-based embeddings perform on par with a specialized EHR foundation model, CLMBR-T-Base, across 15 clinical tasks from the EHRSHOT benchmark. In an external validation using the UK Biobank, an LLM-based model shows statistically significant improvements for some tasks, which we attribute to higher vocabulary coverage and slightly better generalization. Overall, we reveal a trade-off between the computational efficiency of specialized EHR models and the portability and data independence of LLM-based embeddings.