mlsanityData to 14 Oct 2026

AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models

arXiv 2105.00827 ↗First posted 16 Apr 2021cs.CL cs.AI cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

Transformer-based pretrained language models (PLMs) have started a new era in modern natural language processing (NLP). These models combine the power of transformers, transfer learning, and self-supervised learning (SSL). Following the success of these models in the general domain, the biomedical research community has developed various in-domain PLMs starting from BioBERT to the latest BioELECTRA and BioALBERT models. We strongly believe there is a need for a survey paper that can provide a comprehensive survey of various transformer-based biomedical pretrained language models (BPLMs). In this survey, we start with a brief overview of foundational concepts like self-supervised learning, embedding layer and transformer encoder layers. We discuss core concepts of transformer-based PLMs like pretraining methods, pretraining tasks, fine-tuning methods, and various embedding types specific to biomedical domain. We introduce a taxonomy for transformer-based BPLMs and then discuss all the models. We discuss various challenges and present possible solutions. We conclude by highlighting some of the open issues which will drive the research community to further improve transformer-based BPLMs.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 110 of its 221 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

110 of 221 references.

References · 110

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2014Going Deeper with Convolutionscited by 4,241
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  11. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
38 more
  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2018Deep contextualized word representationscited by 2,290
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  9. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  10. 2019Language Models as Knowledge Bases?cited by 1,936
  11. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  13. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  14. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  15. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  16. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  17. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  18. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  19. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  20. 2021Transformers in Vision: A Surveycited by 1,013
  21. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  22. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  23. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  24. 2017A Survey on Multi-Task Learningcited by 840
  25. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  26. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  27. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  28. 2020Self-supervised Learning: Generative or Contrastivecited by 694
  29. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  30. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  31. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  32. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 373
  33. 2019CamemBERT: a Tasty French Language Modelcited by 368
  34. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 362
  35. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  36. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  37. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 299
  38. 2019BEHRT: Transformer for Electronic Health Recordscited by 263

50 most cited of 110 All 110 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 2; 2023: 15; 2024: 8; 2025: 3; 2026: 1.

32 papers.

Sort:

By area:

  1. 2023Knowledge Editing for Large Language Models: A Surveycited by 194
  2. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  3. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  4. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  5. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 71
  6. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  7. 2023Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physicscited by 49
  8. 2023A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Taskscited by 36
  9. 2022Spatial Transformer Network on Skeleton-based Gait Recognitioncited by 33
  10. 2024TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Datacited by 32
  11. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  12. 2023Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformerscited by 22
20 more
  1. 2024Emergency Department Decision Support using Clinical Pseudo-notescited by 22
  2. 2021Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERTcited by 14
  3. 2023ExBEHRT: Extended Transformer for Electronic Health Records to Predict Disease Subtypes & Progressionscited by 12
  4. 2024Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcarecited by 12
  5. 2021Biomedical and Clinical Language Models for Spanish: On the Benefits of Domain-Specific Pretraining in a Mid-Resource Scenariocited by 11
  6. 2023Exploring the In-context Learning Ability of Large Language Model for Biomedical Concept Linkingcited by 11
  7. 2024Fairness Definitions in Language Models Explainedcited by 11
  8. 2023Transformer-based Generative Adversarial Networks in Computer Vision: A Comprehensive Surveycited by 10
  9. 2024Performance of Recent Large Language Models for a Low-Resourced Languagecited by 7
  10. 2022A Comparative Evaluation Of Transformer Models For De-Identification Of Clinical Text Datacited by 5
  11. 2023A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniquescited by 4
  12. 2023A Lightweight Transformer for Faster and Robust EBSD Data Collectioncited by 3
  13. 2025Towards Transparent AI: A Survey on Explainable Language Modelscited by 3
  14. 2023Multilingual Multiword Expression Identification Using Lateral Inhibition and Domain Adaptationcited by 2
  15. 2024A Semantic and Motion-Aware Spatiotemporal Transformer Network for Action Detectioncited by 1
  16. 2024Enhancing Natural Language Inference Performance with Knowledge Graph for COVID-19 Automated Fact-Checking in Indonesian Languagecited by 1
  17. 2024COVIDHealth: A Benchmark Twitter Dataset and Machine Learning based Web Application for Classifying COVID-19 Discussionscited by 0
  18. 2025Language modelling techniques for analysing the impact of human genetic variationcited by 0
  19. 2025Trajectory-guided discharge stratification for heart failure using short-context electronic health record sequence modelingcited by 0
  20. 2026MiNER: Fine-Tuned Biomedical Natural Language Processing for Malaria Disease Entity Recognition in Clinical Textscited by 0