mlsanityData to 7 Oct 2026

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

arXiv 2007.15779 ↗First posted 31 Jul 2020cs.CL cs.LGcited by 996

Not on the map. Why?

Cited by 47 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this paper, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly-available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition (NER). To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB.

Key facts. Cited by 996 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 63; 2022: 121; 2023: 235; 2024: 215; 2025: 214; 2026: 148.

996 papers.

Sort:

By area:

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  3. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  4. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  5. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  6. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  7. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  8. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  9. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  10. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  11. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  12. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
38 more
  1. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  2. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  3. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  4. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  5. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  6. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  7. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  8. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  9. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 206
  10. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  11. 2022Can large language models reason about medical questions?cited by 194
  12. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 182
  13. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 170
  14. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  15. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  16. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
  17. 2021MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcarecited by 159
  18. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  19. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
  20. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 140
  21. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  22. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  23. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 130
  24. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  25. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 128
  26. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  27. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  28. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  29. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  30. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 122
  31. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  32. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  33. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  34. 2023Large Language Models are Few-Shot Health Learnerscited by 115
  35. 2024BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at oncecited by 107
  36. 2022ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reportscited by 100
  37. 2023BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associationscited by 100
  38. 2021CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmarkcited by 99

50 most cited of 996 All 996 (JSON)