# Contextualized Embeddings in Named-Entity Recognition: An Empirical Study on Generalization

arXiv: https://arxiv.org/abs/2001.08053 · HTML: https://mlsanity.com/paper/2001.08053/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jan 2020.

## Built on by (most cited first, 15 of 15)

- [Named Entity Recognition and Classification on Historical Documents: A Survey](https://mlsanity.com/paper/2109.11406/) (2021)
- [Probing Pre-trained Auto-regressive Language Models for Named Entity Typing and Recognition](https://mlsanity.com/paper/2108.11857/) (2021)
- [What do we Really Know about State of the Art NER?](https://mlsanity.com/paper/2205.00034/) (2022)
- [How Do Your Biomedical Named Entity Recognition Models Generalize to Novel Entities?](https://mlsanity.com/paper/2101.00160/) (2021)
- [Benchmarking Advanced Text Anonymisation Methods: A Comparative Study on Novel and Traditional Approaches](https://mlsanity.com/paper/2404.14465/) (2024)
- [Evaluation of contextual embeddings on less-resourced languages](https://mlsanity.com/paper/2107.10614/) (2021)
- [Information Redundancy and Biases in Public Document Information Extraction Benchmarks](https://mlsanity.com/paper/2304.14936/) (2023)
- [Separating Retention from Extraction in the Evaluation of End-to-end Relation Extraction](https://mlsanity.com/paper/2109.12008/) (2021)
- [Investigation on Data Adaptation Techniques for Neural Named Entity Recognition](https://mlsanity.com/paper/2110.05892/) (2021)
- [Data Augmentation for Robust Character Detection in Fantasy Novels](https://mlsanity.com/paper/2302.04555/) (2023)
- [Contextualized Token Discrimination for Speech Search Query Correction](https://mlsanity.com/paper/2509.04393/) (2025)
- [Extracting Impact Model Narratives from Social Services' Text](https://mlsanity.com/paper/2204.09557/) (2022)
- [Dynamic Named Entity Recognition](https://mlsanity.com/paper/2302.10314/) (2023)
- [Text Data Integration](https://mlsanity.com/paper/2603.27055/) (2026)
- [On-Device Named-Entity Recognition: A Deployability Study of Accuracy, Cost, Reliability, and Confidence](https://mlsanity.com/paper/2610.00007/) (2026)

## Abstract (arXiv)

Contextualized embeddings use unsupervised language model pretraining to compute word representations depending on their context. This is intuitively useful for generalization, especially in Named-Entity Recognition where it is crucial to detect mentions never seen during training. However, standard English benchmarks overestimate the importance of lexical over contextual features because of an unrealistic lexical overlap between train and test mentions. In this paper, we perform an empirical analysis of the generalization capabilities of state-of-the-art contextualized embeddings by separating mentions by novelty and with out-of-domain evaluation. We show that they are particularly beneficial for unseen mentions detection, especially out-of-domain. For models trained on CoNLL03, language model contextualization leads to a +1.2% maximal relative micro-F1 score increase in-domain against +13% out-of-domain on the WNUT dataset