# From Zero to Hero: Harnessing Transformers for Biomedical Named Entity Recognition in Zero- and Few-shot Contexts

arXiv: https://arxiv.org/abs/2305.04928 · HTML: https://mlsanity.com/paper/2305.04928/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 64 references that are in the map.
- First posted on arXiv 5 May 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [A Survey on Deep Learning for Named Entity Recognition](https://mlsanity.com/paper/1812.09449/) (2018)
- [Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring](https://mlsanity.com/paper/1905.01969/) (2019)
- [A Survey on Recent Advances in Named Entity Recognition from Deep Learning models](https://mlsanity.com/paper/1910.11470/) (2019)
- [SciFive: a text-to-text transformer model for biomedical literature](https://mlsanity.com/paper/2106.03598/) (2021)
- [BioRED: A Rich Biomedical Relation Extraction Dataset](https://mlsanity.com/paper/2204.04263/) (2022)
- [Few-shot classification in Named Entity Recognition Task](https://mlsanity.com/paper/1812.06158/) (2018)
- [Few-Shot Named Entity Recognition: A Comprehensive Study](https://mlsanity.com/paper/2012.14978/) (2020)
- [Few-shot Learning for Named Entity Recognition in Medical Text](https://mlsanity.com/paper/1811.05468/) (2018)
- [QaNER: Prompting Question Answering Models for Few-shot Named Entity Recognition](https://mlsanity.com/paper/2203.01543/) (2022)
- [Robust Multilingual Named Entity Recognition with Shallow Semi-Supervised Features](https://mlsanity.com/paper/1701.09123/) (2017)
- [Comparison of biomedical relationship extraction methods and models for knowledge graph creation](https://mlsanity.com/paper/2201.01647/) (2022)
- [Medical Coding with Biomedical Transformer Ensembles and Zero/Few-shot Learning](https://mlsanity.com/paper/2206.02662/) (2022)
- [MASK: A flexible framework to facilitate de-identification of clinical texts](https://mlsanity.com/paper/2005.11687/) (2020)
- [Zero-Shot Learning in Named-Entity Recognition with External Knowledge](https://mlsanity.com/paper/2111.07734/) (2021)

## Built on by (most cited first, 2 of 2)

- [Novel Benchmark for NER in the Wastewater and Stormwater Domain](https://mlsanity.com/paper/2506.01938/) (2025)
- [Few-Label Multimodal Modeling of SNP Variants and ECG Phenotypes Using Large Language Models for Cardiovascular Risk Stratification](https://mlsanity.com/paper/2510.16536/) (2025)

## Abstract (arXiv)

Supervised named entity recognition (NER) in the biomedical domain depends on large sets of annotated texts with the given named entities. The creation of such datasets can be time-consuming and expensive, while extraction of new entities requires additional annotation tasks and retraining the model. To address these challenges, this paper proposes a method for zero- and few-shot NER in the biomedical domain. The method is based on transforming the task of multi-class token classification into binary token classification and pre-training on a large amount of datasets and biomedical entities, which allow the model to learn semantic relations between the given and potentially novel named entity labels. We have achieved average F1 scores of 35.44% for zero-shot NER, 50.10% for one-shot NER, 69.94% for 10-shot NER, and 79.51% for 100-shot NER on 9 diverse evaluated biomedical entities with fine-tuned PubMedBERT-based model. The results demonstrate the effectiveness of the proposed method for recognizing new biomedical entities with no or limited number of examples, outperforming previous transformer-based methods, and being comparable to GPT3-based models using models with over 1000 times fewer parameters. We make models and developed code publicly available.