mlsanityData to 7 Oct 2026

Large Language Models are Powerful Electronic Health Record Encoders

arXiv 2502.17403 ↗First posted 24 Feb 2025cs.LG cs.AI cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

Electronic Health Records (EHRs) offer considerable potential for clinical prediction, but their complexity and heterogeneity challenge traditional machine learning. Domain-specific EHR foundation models trained on unlabeled EHR data have shown improved predictive accuracy and generalization. However, their development is constrained by limited data access and site-specific vocabularies. We convert EHR data into plain text by replacing medical codes with natural-language descriptions, enabling general-purpose Large Language Models (LLMs) to produce high-dimensional embeddings for downstream prediction tasks without access to private medical training data. LLM-based embeddings perform on par with a specialized EHR foundation model, CLMBR-T-Base, across 15 clinical tasks from the EHRSHOT benchmark. In an external validation using the UK Biobank, an LLM-based model shows statistically significant improvements for some tasks, which we attribute to higher vocabulary coverage and slightly better generalization. Overall, we reveal a trade-off between the computational efficiency of specialized EHR models and the portability and data independence of LLM-based embeddings.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 36 of its 80 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

36 of 80 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  8. 2019Publicly Available Clinical BERT Embeddingscited by 995
  9. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  10. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  11. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  12. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
24 more
  1. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 368
  2. 2018Scalable and accurate deep learning for electronic health recordscited by 275
  3. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  4. 2019BEHRT: Transformer for Electronic Health Recordscited by 258
  5. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  6. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 233
  7. 2024Generative Representational Instruction Tuningcited by 227
  8. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 206
  9. 2023Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Studycited by 116
  10. 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103
  11. 2023EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Modelscited by 101
  12. 2020Language Models Are An Effective Patient Representation Learning Technique For Electronic Health Record Datacited by 85
  13. 2021CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction taskscited by 75
  14. 2024Repetition Improves Language Model Embeddingscited by 70
  15. 2023CPLLM: Clinical Prediction with Large Language Modelscited by 34
  16. 2024ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?cited by 31
  17. 2024Prompting Large Language Models for Zero-Shot Clinical Prediction with Structured Longitudinal Electronic Health Record Datacited by 22
  18. 2024CORE-BEHRT: A Carefully Optimized and Rigorously Evaluated BEHRTcited by 20
  19. 2019Early detection of sepsis utilizing deep learning on electronic health record event sequencescited by 15
  20. 2024LLMs-based Few-Shot Disease Predictions using EHR: A Novel Approach Combining Predictive Agent Reasoning and Critical Agent Instructioncited by 11
  21. 2024When Raw Data Prevails: Are Large Language Model Embeddings Effective in Numerical Data Representation for Medical Machine Learning Applications?cited by 11
  22. 2023Retrieving Evidence from EHRs with LLMs: Possibilities and Challengescited by 8
  23. 2023Clinical Risk Prediction Using Language Models: Benefits And Considerationscited by 6
  24. 2025Memorize and Rank: Elevating Large Language Models for Clinical Diagnosis Predictioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 18.

25 papers.

Sort:

By area:

  1. 2025EHR-R1: A Reasoning-Enhanced Foundational Language Model for Electronic Health Record Analysiscited by 17
  2. 2025ChronoFormer: Time-Aware Transformer Architectures for Structured Clinical Event Modelingcited by 11
  3. 2026EHR-RAG: Bridging Long-Horizon Structured Electronic Health Records and Large Language Models via Enhanced Retrieval-Augmented Generationcited by 6
  4. 2026One Loss to Rule Them All: Marked Time-to-Event for Structured EHR Foundation Modelscited by 5
  5. 2025Fusion or Confusion? Multimodal Complexity Is Not All You Needcited by 4
  6. 2026AgentEHR: Advancing Autonomous Clinical Decision-Making via Retrospective Summarizationcited by 4
  7. 2025Rethinking Tokenization for Clinical Time Series: When Less is Morecited by 3
  8. 2026LLMs can construct powerful representations and streamline sample-efficient supervised learningcited by 2
  9. 2026PORTER: Language-Grounded Event Representations for Portable Structured EHR Foundation Modelscited by 2
  10. 2025The CRITICAL Records Integrated Standardization Pipeline (CRISP): End-to-End Processing of Large-scale Multi-institutional OMOP CDM Datacited by 1
  11. 2026PaReGTA: A Temporally Aware LLM-Based Patient Representation Framework for EHR Analyticscited by 1
  12. 2026Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learningcited by 1
13 more
  1. 2025APRIL: Annotations for Policy evaluation with Reliable Inference from LLMscited by 0
  2. 2025Aligning Probabilistic Beliefs under Informative Missingness: LLM Steerability in Clinical Reasoningcited by 0
  3. 2026CacheMind: From Miss Rates to Why -- Natural-Language, Trace-Grounded Reasoning for Cache Replacementcited by 0
  4. 2026MATA-Former & SIICU: Semantic Aware Temporal Alignment for High-Fidelity ICU Risk Predictioncited by 0
  5. 2026Representation Before Training: A Practical Benchmark for Generative Medical Event Model Tokenizationcited by 0
  6. 2026OphthaDT: Generative Digital Twins for Forecasting Visual Acuity Trajectories in Ophthalmologycited by 0
  7. 2026A General Multimodal Probing Framework for Data Fusion and Prediction with Frozen Large Language Models: Evidence from Multimodal EHR Datacited by 0
  8. 2026Large Language Models as Unified Multimodal Learners for Clinical Predictioncited by 0
  9. 2026Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learningcited by 0
  10. 2026Patients-like-me: A Variational LM--GNN Framework for Explainable Clinical Predictioncited by 0
  11. 2026Foresight-England: Development of a National-Scale Generative AI Model of Electronic Health Records for Medical Event Prediction across the COVID-19 Pandemiccited by 0
  12. 2026EHRAdapt: Adapting Pretrained Language Models to Electronic Health Records with Semantic Priors for Rare Clinical Eventscited by 0
  13. 2026GLoC-EHR: Evidence-Cited Clinical Reasoning over Global Context and Local EHR Eventscited by 0