mlsanityData to 7 Oct 2026

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

arXiv 2110.05006 ↗First posted 11 Oct 2021cs.CLcited by 87

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (PLMs) have been the de facto paradigm for most natural language processing (NLP) tasks. This also benefits biomedical domain: researchers from informatics, medicine, and computer science (CS) communities propose various PLMs trained on biomedical datasets, e.g., biomedical text, electronic health records, protein, and DNA sequences for various biomedical tasks. However, the cross-discipline characteristics of biomedical PLMs hinder their spreading among communities; some existing works are isolated from each other without comprehensive comparison and discussions. It expects a survey that not only systematically reviews recent advances of biomedical PLMs and their applications but also standardizes terminology and benchmarks. In this paper, we summarize the recent progress of pre-trained language models in the biomedical domain and their applications in biomedical downstream tasks. Particularly, we discuss the motivations and propose a taxonomy of existing biomedical PLMs. Their applications in biomedical downstream tasks are exhaustively discussed. At last, we illustrate various limitations and future trends, which we hope can provide inspiration for the future research of the research community.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 187 of its 410 references that are in the map. First posted on arXiv 11 Oct 2021.

Builds on

187 of 410 references.

References · 187

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  11. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  13. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  14. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  15. 2018Deep contextualized word representationscited by 2,285
  16. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  17. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  18. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  19. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  20. 2019Language Models as Knowledge Bases?cited by 1,922
  21. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  22. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  23. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  24. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  25. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  26. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  27. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  28. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  29. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  30. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  31. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  32. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  33. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  34. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  35. 2019Publicly Available Clinical BERT Embeddingscited by 995
  36. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  37. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  38. 2020Efficient Transformers: A Surveycited by 784

50 most cited of 187 All 187 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 21; 2024: 35; 2025: 21; 2026: 3.

87 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  2. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  3. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  4. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  5. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  6. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 91
  7. 2023DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultationcited by 89
  8. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  9. 2023K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilizationcited by 69
  10. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 57
  11. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  12. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
38 more
  1. 2023A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Taskscited by 36
  2. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  3. 2024ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?cited by 31
  4. 2024Libra: Leveraging Temporal Images for Biomedical Radiology Analysiscited by 29
  5. 2024Demographic Bias of Expert-Level Vision-Language Foundation Models in Medical Imagingcited by 24
  6. 2021Transformer Uncertainty Estimation with Hierarchical Stochastic Attentioncited by 21
  7. 2022Readability Controllable Biomedical Document Summarizationcited by 19
  8. 2025Protein Large Language Models: A Comprehensive Surveycited by 18
  9. 2022Protein Language Models and Structure Prediction: Connection and Progressioncited by 17
  10. 2023Transformers in Healthcare: A Surveycited by 17
  11. 2022In-BoXBART: Get Instructions into Biomedical Multi-Task Learningcited by 16
  12. 2022LingYi: Medical Conversational Question Answering System based on Multi-modal Knowledge Graphscited by 15
  13. 2024Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Biascited by 15
  14. 2024PromptLink: Leveraging Large Language Models for Cross-Source Biomedical Concept Linkingcited by 15
  15. 2024Towards Reliable Medical Question Answering: Techniques and Challenges in Mitigating Hallucinations in Language Modelscited by 15
  16. 2023Clinical Concept and Relation Extraction Using Prompt-based Machine Reading Comprehensioncited by 13
  17. 2022GRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarizationcited by 10
  18. 2024A dataset and benchmark for hospital course summarization with adapted large language modelscited by 10
  19. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  20. 2025Vision Foundation Models in Medical Image Analysis: Advances and Challengescited by 9
  21. 2025Uncertainty-Aware Large Language Models for Explainable Disease Diagnosiscited by 9
  22. 2025HypKG: Hypergraph-based Knowledge Graph Contextualization for Precision Healthcarecited by 9
  23. 2023Investigating Large Language Models and Control Mechanisms to Improve Text Readability of Biomedical Abstractscited by 8
  24. 2024HunFlair2 in a cross-corpus evaluation of biomedical named entity recognition and normalization toolscited by 8
  25. 2023BioLORD-2023: Semantic Textual Representations Fusing LLM and Clinical Knowledge Graph Insightscited by 7
  26. 2024Time Matters: Examine Temporal Effects on Biomedical Language Modelscited by 7
  27. 2024MedINST: Meta Dataset of Biomedical Instructionscited by 7
  28. 2023FactReranker: Fact-guided Reranker for Faithful Radiology Report Summarizationcited by 6
  29. 2023Collaborative Synthesis of Patient Records through Multi-Visit Health State Inferencecited by 6
  30. 2024Clinical Insights: A Comprehensive Review of Language Models in Medicinecited by 6
  31. 2024SimSUM: Simulated Benchmark with Structured and Unstructured Medical Recordscited by 6
  32. 2022Transformers and the representation of biomedical background knowledgecited by 5
  33. 2023Interactive Molecular Discovery with Natural Languagecited by 5
  34. 2025MoodAngels: A Retrieval-augmented Multi-agent Framework for Psychiatry Diagnosiscited by 5
  35. 2023Extensive Evaluation of Transformer-based Architectures for Adverse Drug Events Extractioncited by 4
  36. 2024Autocompletion of Chief Complaints in the Electronic Health Records using Large Language Modelscited by 4
  37. 2024RAmBLA: A Framework for Evaluating the Reliability of LLMs as Assistants in the Biomedical Domaincited by 4
  38. 2024BioMistral-NLU: Towards More Generalizable Medical Language Understanding through Instruction Tuningcited by 4

50 most cited of 87 All 87 (JSON)