mlsanityData to 7 Oct 2026

Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction

arXiv 2005.12833 ↗First posted 22 May 2020cs.CL cs.LG cs.NEcited by 368

Not on the map. Why?

Abstract (arXiv)

Deep learning (DL) based predictive models from electronic health records (EHR) deliver impressive performance in many clinical tasks. Large training cohorts, however, are often required to achieve high accuracy, hindering the adoption of DL-based models in scenarios with limited training data size. Recently, bidirectional encoder representations from transformers (BERT) and related models have achieved tremendous successes in the natural language processing domain. The pre-training of BERT on a very large training corpus generates contextualized embeddings that can boost the performance of models trained on smaller datasets. We propose Med-BERT, which adapts the BERT framework for pre-training contextualized embedding models on structured diagnosis data from 28,490,650 patients EHR dataset. Fine-tuning experiments are conducted on two disease-prediction tasks: (1) prediction of heart failure in patients with diabetes and (2) prediction of pancreatic cancer from two clinical databases. Med-BERT substantially improves prediction accuracy, boosting the area under receiver operating characteristics curve (AUC) by 2.02-7.12%. In particular, pre-trained Med-BERT substantially improves the performance of tasks with very small fine-tuning training sets (300-500 samples) boosting the AUC by more than 20% or equivalent to the AUC of 10 times larger training set. We believe that Med-BERT will benefit disease-prediction studies with small local training datasets, reduce data collection expenses, and accelerate the pace of artificial intelligence aided healthcare.

Key facts. Cited by 368 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 40; 2023: 60; 2024: 73; 2025: 79; 2026: 89.

368 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  3. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  4. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  5. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  6. 2023Auditing large language models: a three-layered approachcited by 135
  7. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  8. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  9. 2023EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Modelscited by 101
  10. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  11. 2021ClimateBert: A Pretrained Language Model for Climate-Related Textcited by 89
  12. 2024ECG-FM: An Open Electrocardiogram Foundation Modelcited by 89
38 more
  1. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  2. 2021CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction taskscited by 75
  3. 2023MOTOR: A Time-To-Event Foundation Model For Structured Medical Recordscited by 75
  4. 2022Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequencescited by 67
  5. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  6. 2023A Bibliometric Review of Large Language Models Research from 2017 to 2023cited by 56
  7. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  8. 2021Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health recordscited by 53
  9. 2023Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Eventscited by 46
  10. 2022A Multifaceted Benchmarking of Synthetic Electronic Health Record Generation Modelscited by 43
  11. 2023ChatGPT for Shaping the Future of Dentistry: The Potential of Multi-Modal Large Language Modelcited by 40
  12. 2024Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRscited by 39
  13. 2021Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Reviewcited by 35
  14. 2022GenHPF: General Healthcare Predictive Framework with Multi-task Multi-source Learningcited by 35
  15. 2023CPLLM: Clinical Prediction with Large Language Modelscited by 34
  16. 2024Large Language Models in Cybersecurity: State-of-the-Artcited by 34
  17. 2021AMMU : A Survey of Transformer-based Biomedical Pretrained Language Modelscited by 32
  18. 2024EHRMamba: Towards Generalizable and Scalable Foundation Models for Electronic Health Recordscited by 31
  19. 2023A Multi-Center Study on the Adaptability of a Shared Foundation Model for Electronic Health Recordscited by 30
  20. 2023The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRscited by 29
  21. 2024MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosiscited by 28
  22. 2022Handling missing values in healthcare data: A systematic review of deep learning-based imputation techniquescited by 25
  23. 2023BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processingcited by 25
  24. 2025Large Language Models are Powerful Electronic Health Record Encoderscited by 25
  25. 2021MedGPT: Medical Concept Prediction from Clinical Narrativescited by 24
  26. 2021Unifying Heterogeneous Electronic Health Records Systems via Text-Based Code Embeddingcited by 24
  27. 2025Understanding the Uncertainty of LLM Explanations: A Perspective Based on Reasoning Topologycited by 24
  28. 2021RoBERTuito: a pre-trained language model for social media text in Spanishcited by 22
  29. 2023AI vs. Human -- Differentiation Analysis of Scientific Content Generationcited by 21
  30. 2022Write It Like You See It: Detectable Differences in Clinical Notes By Race Lead To Differential Model Recommendationscited by 20
  31. 2023Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Reviewcited by 20
  32. 2024Leveraging Large Language Model as Simulated Patients for Clinical Educationcited by 20
  33. 2024CORE-BEHRT: A Carefully Optimized and Rigorously Evaluated BEHRTcited by 20
  34. 2022On the Transformation of Latent Space in Fine-Tuned NLP Modelscited by 19
  35. 2023DuETT: Dual Event Time Transformer for Electronic Health Recordscited by 17
  36. 2023Transformers in Healthcare: A Surveycited by 17
  37. 2023Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPTcited by 17
  38. 2024Health-LLM: Personalized Retrieval-Augmented Disease Prediction Systemcited by 17

50 most cited of 368 All 368 (JSON)