mlsanityData to 7 Oct 2026

Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERT

arXiv 2107.04374 ↗First posted 9 Jul 2021cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

The availability of biomedical text data and advances in natural language processing (NLP) have made new applications in biomedical NLP possible. Language models trained or fine tuned using domain specific corpora can outperform general models, but work to date in biomedical NLP has been limited in terms of corpora and tasks. We present BioALBERT, a domain-specific adaptation of A Lite Bidirectional Encoder Representations from Transformers (ALBERT), trained on biomedical (PubMed and PubMed Central) and clinical (MIMIC-III) corpora and fine tuned for 6 different tasks across 20 benchmark datasets. Experiments show that BioALBERT outperforms the state of the art on named entity recognition (+11.09% BLURB score improvement), relation extraction (+0.80% BLURB score), sentence similarity (+1.05% BLURB score), document classification (+0.62% F1-score), and question answering (+2.83% BLURB score). It represents a new state of the art in 17 out of 20 benchmark datasets. By making BioALBERT models and data available, our aim is to help the biomedical NLP community avoid computational costs of training and establish a new set of baselines for future efforts across a broad range of biomedical NLP tasks.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 18 of its 42 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

18 of 42 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  4. 2018Deep contextualized word representationscited by 2,285
  5. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  6. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  7. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  8. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  9. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 149
  10. 2016An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wildcited by 146
  11. 2019Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approachescited by 64
  12. 2019Enhancing Clinical Concept Extraction with Contextual Embeddingscited by 48
6 more
  1. 2018MedSTS: A Resource for Clinical Semantic Textual Similaritycited by 39
  2. 2019Probing Biomedical Embeddings from Language Modelscited by 39
  3. 2021Improving Biomedical Pretrained Language Models with Knowledgecited by 36
  4. 2021AMMU : A Survey of Transformer-based Biomedical Pretrained Language Modelscited by 32
  5. 2020Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QAcited by 23
  6. 2020BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognitioncited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 5; 2025: 2; 2026: 2.

14 papers.

Sort:

  1. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  2. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  3. 2022Benchmark datasets driving artificial intelligence development fail to capture the needs of medical professionalscited by 16
  4. 2023Investigating Large Language Models and Control Mechanisms to Improve Text Readability of Biomedical Abstractscited by 8
  5. 2022Benchmarking for Public Health Surveillance tasks on Social Media with a Domain-Specific Pretrained Language Modelcited by 7
  6. 2024Fine-Tuning Large Language Models for Scientific Text Classification: A Comparative Studycited by 5
  7. 2024RAmBLA: A Framework for Evaluating the Reliability of LLMs as Assistants in the Biomedical Domaincited by 4
  8. 2024Question answering systems for health professionals at the point of care -- a systematic reviewcited by 3
  9. 2025Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Reviewcited by 3
  10. 2023Soft-prompt tuning to predict lung cancer using primary care free-text Dutch medical notescited by 2
  11. 2024Large Language Models for Biomedical Text Simplification: Promising But Not There Yetcited by 1
  12. 2025Advancing Scientific Text Classification: Fine-Tuned Models with Dataset Expansion and Hard-Votingcited by 0
2 more
  1. 2026EQ-5D Classification Using Biomedical Entity-Enriched Pre-trained Language Models and Multiple Instance Learningcited by 0
  2. 2026Ensembles of Large Language Models for Identifying EQ-5D Studies in PubMed Based on Their Abstractscited by 0