mlsanityData to 14 Oct 2026

Serialized EHR make for good text representations

arXiv 2510.13843 ↗First posted 11 Oct 2025cs.CL cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

The emergence of foundation models in healthcare has opened new avenues for learning generalizable representations from large scale clinical data. Yet, existing approaches often struggle to reconcile the tabular and event based nature of Electronic Health Records (EHRs) with the sequential priors of natural language models. This structural mismatch limits their ability to capture longitudinal dependencies across patient encounters. We introduce SerialBEHRT, a domain aligned foundation model that extends SciBERT through additional pretraining on structured EHR sequences. SerialBEHRT is designed to encode temporal and contextual relationships among clinical events, thereby producing richer patient representations. We evaluate its effectiveness on the task of antibiotic susceptibility prediction, a clinically meaningful problem in antibiotic stewardship. Through extensive benchmarking against state of the art EHR representation strategies, we demonstrate that SerialBEHRT achieves superior and more consistent performance, highlighting the importance of temporal serialization in foundation model pretraining for healthcare.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 43 of its 73 references that are in the map. First posted on arXiv 11 Oct 2025.

Builds on

43 of 73 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  8. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  9. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  10. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  11. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  12. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
31 more
  1. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  2. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  3. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
  4. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 398
  5. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 373
  6. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  7. 2019BEHRT: Transformer for Electronic Health Recordscited by 263
  8. 2022A Survey on Generative Diffusion Modelcited by 240
  9. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  10. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  11. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 148
  12. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 148
  13. 2018Construction of the Literature Graph in Semantic Scholarcited by 118
  14. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 106
  15. 2023EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Modelscited by 105
  16. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 102
  17. 2019Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answeringcited by 96
  18. 2020Fast WordPiece Tokenizationcited by 87
  19. 2023MOTOR: A Time-To-Event Foundation Model For Structured Medical Recordscited by 78
  20. 2020BioMegatron: Larger Biomedical Domain Language Modelcited by 76
  21. 2021CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction taskscited by 76
  22. 2021Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health recordscited by 54
  23. 2020A Label Attention Model for ICD Coding from Clinical Textcited by 48
  24. 2023A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnosticscited by 43
  25. 2023Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicinecited by 24
  26. 2024Can Large Language Models abstract Medical Coded Language?cited by 20
  27. 2024Text Serialization and Their Relationship with the Conventional Paradigms of Tabular Machine Learningcited by 17
  28. 2025A Case Study Exploring the Current Landscape of Synthetic Medical Record Generation with Commercial LLMscited by 13
  29. 2023ExBEHRT: Extended Transformer for Electronic Health Records to Predict Disease Subtypes & Progressionscited by 12
  30. 2023Towards Better Serialization of Tabular Data for Few-shot Classification with Large Language Modelscited by 12
  31. 2024Enhancing Antibiotic Stewardship using a Natural Language Approach for Better Feature Representationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Learning Clinical Representations Under Systematic Distribution Shiftcited by 2
  2. 2026Discriminative Representation Learning for Clinical Predictioncited by 2
  3. 2026Learning Longitudinal Health Representations from EHR and Wearable Datacited by 1
  4. 2026Uncertainty-Aware Foundation Models for Clinical Datacited by 1
  5. 2026Event Fields: Learning Latent Event Structure for Waveform Foundation Modelscited by 0
  6. 2026WISTERIA: Learning Clinical Representations from Noisy Supervision via Multi-View Consistency in Electronic Health Recordscited by 0
  7. 2026AURORA: Contextual Orthogonalization for Geometric Representation Learning in Healthcare Foundation Modelscited by 0