# Preserving Empirical Probabilities in BERT for Small-sample Clinical Entity Recognition

arXiv: https://arxiv.org/abs/2409.03238 · HTML: https://mlsanity.com/paper/2409.03238/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 5 Sep 2024.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [BinaryBERT: Pushing the Limit of BERT Quantization](https://mlsanity.com/paper/2012.15701/) (2020)
- [Few-NERD: A Few-Shot Named Entity Recognition Dataset](https://mlsanity.com/paper/2105.07464/) (2021)
- [Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review](https://mlsanity.com/paper/2107.02975/) (2021)
- [Benchmark and Best Practices for Biomedical Knowledge Graph Embeddings](https://mlsanity.com/paper/2006.13774/) (2020)
- [Clinical Named Entity Recognition using Contextualized Token Representations](https://mlsanity.com/paper/2106.12608/) (2021)
- [KG-MTT-BERT: Knowledge Graph Enhanced BERT for Multi-Type Medical Text Classification](https://mlsanity.com/paper/2210.03970/) (2022)

## Abstract (arXiv)

Named Entity Recognition (NER) encounters the challenge of unbalanced labels, where certain entity types are overrepresented while others are underrepresented in real-world datasets. This imbalance can lead to biased models that perform poorly on minority entity classes, impeding accurate and equitable entity recognition. This paper explores the effects of unbalanced entity labels of the BERT-based pre-trained model. We analyze the different mechanisms of loss calculation and loss propagation for the task of token classification on randomized datasets. Then we propose ways to improve the token classification for the highly imbalanced task of clinical entity recognition.