# BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognition

arXiv: https://arxiv.org/abs/2009.09223 · HTML: https://mlsanity.com/paper/2009.09223/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2020.

## Built on by (most cited first, 19 of 19)

- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review](https://mlsanity.com/paper/2107.02975/) (2021)
- [AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models](https://mlsanity.com/paper/2105.00827/) (2021)
- [Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognition](https://mlsanity.com/paper/2307.10757/) (2023)
- [Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERT](https://mlsanity.com/paper/2107.04374/) (2021)
- [Localising In-Domain Adaptation of Transformer-Based Biomedical Language Models](https://mlsanity.com/paper/2212.10422/) (2022)
- [Investigating Large Language Models and Control Mechanisms to Improve Text Readability of Biomedical Abstracts](https://mlsanity.com/paper/2309.13202/) (2023)
- [Text Mining of Stocktwits Data for Predicting Stock Prices](https://mlsanity.com/paper/2103.16388/) (2021)
- [CLIN-X: pre-trained language models and a study on cross-task transfer for concept extraction in the clinical domain](https://mlsanity.com/paper/2112.08754/) (2021)
- [Benchmarking for Public Health Surveillance tasks on Social Media with a Domain-Specific Pretrained Language Model](https://mlsanity.com/paper/2204.04521/) (2022)
- [Time Matters: Examine Temporal Effects on Biomedical Language Models](https://mlsanity.com/paper/2407.17638/) (2024)
- [From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine](https://mlsanity.com/paper/2502.09242/) (2025)
- [Classifying vaccine sentiment tweets by modelling domain-specific representation and commonsense knowledge into context-aware attentive GRU](https://mlsanity.com/paper/2106.09589/) (2021)
- [BioADAPT-MRC: Adversarial Learning-based Domain Adaptation Improves Biomedical Machine Reading Comprehension Task](https://mlsanity.com/paper/2202.13174/) (2022)
- [EriBERTa: A Bilingual Pre-Trained Language Model for Clinical Natural Language Processing](https://mlsanity.com/paper/2306.07373/) (2023)
- [Large Language Models for Biomedical Text Simplification: Promising But Not There Yet](https://mlsanity.com/paper/2408.03871/) (2024)
- [DeviceBERT: Applied Transfer Learning With Targeted Annotations and Vocabulary Enrichment to Identify Medical Device and Component Terminology in FDA Recall Summaries](https://mlsanity.com/paper/2406.05307/) (2024)
- [Towards understanding evolution of science through language model series](https://mlsanity.com/paper/2409.09636/) (2024)

## Abstract (arXiv)

In recent years, with the growing amount of biomedical documents, coupled with advancement in natural language processing algorithms, the research on biomedical named entity recognition (BioNER) has increased exponentially. However, BioNER research is challenging as NER in the biomedical domain are: (i) often restricted due to limited amount of training data, (ii) an entity can refer to multiple types and concepts depending on its context and, (iii) heavy reliance on acronyms that are sub-domain specific. Existing BioNER approaches often neglect these issues and directly adopt the state-of-the-art (SOTA) models trained in general corpora which often yields unsatisfactory results. We propose biomedical ALBERT (A Lite Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) bioALBERT, an effective domain-specific language model trained on large-scale biomedical corpora designed to capture biomedical context-dependent NER. We adopted a self-supervised loss used in ALBERT that focuses on modelling inter-sentence coherence to better learn context-dependent representations and incorporated parameter reduction techniques to lower memory consumption and increase the training speed in BioNER. In our experiments, BioALBERT outperformed comparative SOTA BioNER models on eight biomedical NER benchmark datasets with four different entity types. We trained four different variants of BioALBERT models which are available for the research community to be used in future research.