# Clinical Named Entity Recognition using Contextualized Token Representations

arXiv: https://arxiv.org/abs/2106.12608 · HTML: https://mlsanity.com/paper/2106.12608/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 39 references that are in the map.
- First posted on arXiv 23 Jun 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [Scalable and accurate deep learning for electronic health records](https://mlsanity.com/paper/1801.07860/) (2018)
- [Cross-type Biomedical Named Entity Recognition with Deep Multi-Task Learning](https://mlsanity.com/paper/1801.09851/) (2018)
- [Clinical Temporal Relation Extraction with Probabilistic Soft Logic Regularization and Global Inference](https://mlsanity.com/paper/2012.08790/) (2020)
- [Domain Knowledge Empowered Structured Neural Net for End-to-End Event Temporal Relation Extraction](https://mlsanity.com/paper/2009.07373/) (2020)
- [CREATe: Clinical Report Extraction and Annotation Technology](https://mlsanity.com/paper/2103.00562/) (2021)
- [Fast and Accurate Recognition of Chinese Clinical Named Entities with Residual Dilated Convolutions](https://mlsanity.com/paper/1808.08669/) (2018)

## Built on by (most cited first, 4 of 4)

- [Performant ASR Models for Medical Entities in Accented Speech](https://mlsanity.com/paper/2406.12387/) (2024)
- [ExKG-LLM: Leveraging Large Language Models for Automated Expansion of Cognitive Neuroscience Knowledge Graphs](https://mlsanity.com/paper/2503.06479/) (2025)
- [Leveraging Foundation Models for Clinical Text Analysis](https://mlsanity.com/paper/2303.13314/) (2023)
- [Preserving Empirical Probabilities in BERT for Small-sample Clinical Entity Recognition](https://mlsanity.com/paper/2409.03238/) (2024)

## Abstract (arXiv)

The clinical named entity recognition (CNER) task seeks to locate and classify clinical terminologies into predefined categories, such as diagnostic procedure, disease disorder, severity, medication, medication dosage, and sign symptom. CNER facilitates the study of side-effect on medications including identification of novel phenomena and human-focused information extraction. Existing approaches in extracting the entities of interests focus on using static word embeddings to represent each word. However, one word can have different interpretations that depend on the context of the sentences. Evidently, static word embeddings are insufficient to integrate the diverse interpretation of a word. To overcome this challenge, the technique of contextualized word embedding has been introduced to better capture the semantic meaning of each word based on its context. Two of these language models, ELMo and Flair, have been widely used in the field of Natural Language Processing to generate the contextualized word embeddings on domain-generic documents. However, these embeddings are usually too general to capture the proximity among vocabularies of specific domains. To facilitate various downstream applications using clinical case reports (CCRs), we pre-train two deep contextualized language models, Clinical Embeddings from Language Model (C-ELMo) and Clinical Contextual String Embeddings (C-Flair) using the clinical-related corpus from the PubMed Central. Explicit experiments show that our models gain dramatic improvements compared to both static word embeddings and domain-generic language models.