mlsanityData to 7 Oct 2026

Publicly Available Clinical BERT Embeddings

arXiv 1904.03323 ↗First posted 6 Apr 2019cs.CLcited by 995

Not on the map. Why?

Cited by 60 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contextual word embedding models such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) have dramatically improved performance for many natural language processing (NLP) tasks in recent months. However, these models have been minimally explored on specialty corpora, such as clinical text; moreover, in the clinical domain, no publicly-available pre-trained BERT models yet exist. In this work, we address this need by exploring and releasing BERT models for clinical text: one for generic clinical text and another for discharge summaries specifically. We demonstrate that using a domain-specific model yields performance improvements on three common clinical NLP tasks as compared to nonspecific embeddings. These domain-specific models are not as performant on two clinical de-identification tasks, and argue that this is a natural consequence of the differences between de-identified source text and synthetically non de-identified task text.

Key facts. Cited by 995 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 117; 2022: 155; 2023: 208; 2024: 185; 2025: 191; 2026: 139.

995 papers.

Sort:

By area:

  1. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  2. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  3. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  4. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  5. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  6. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 206
  7. 2023Continual Pre-training of Language Modelscited by 194
  8. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 183
  9. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 182
  10. 2021MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcarecited by 159
  11. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  12. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
38 more
  1. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  2. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  3. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 130
  4. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  5. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  6. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  7. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
  8. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  9. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 115
  10. 2023Do We Still Need Clinical Language Models?cited by 97
  11. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Trainingcited by 96
  12. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  13. 2022Improving Chest X-Ray Report Generation by Leveraging Warm Startingcited by 93
  14. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 91
  15. 2021Unified Named Entity Recognition as Word-Word Relation Classificationcited by 89
  16. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  17. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  18. 2022Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Againcited by 83
  19. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  20. 2021MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extractioncited by 82
  21. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  22. 2023Towards Mitigating Hallucination in Large Language Models via Self-Reflectioncited by 81
  23. 2023CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-trainingcited by 81
  24. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  25. 2021Differential Privacy for Text Analytics via Natural Text Sanitizationcited by 74
  26. 2022Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligencecited by 70
  27. 2022Integrated multimodal artificial intelligence framework for healthcare applicationscited by 69
  28. 2021Knowledge Matters: Radiology Report Generation with General and Specific Knowledgecited by 68
  29. 2022Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequencescited by 67
  30. 2022Recovering Private Text in Federated Learning of Language Modelscited by 67
  31. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  32. 2022ET-BERT: A Contextualized Datagram Representation with Pre-training Transformers for Encrypted Traffic Classificationcited by 66
  33. 2024Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancementcited by 66
  34. 2023Frozen Language Model Helps ECG Zero-Shot Learningcited by 62
  35. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  36. 2022SecureBERT: A Domain-Specific Language Model for Cybersecuritycited by 60
  37. 2023LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Modelscited by 58
  38. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 58

50 most cited of 995 All 995 (JSON)