# Generalisation in Named Entity Recognition: A Quantitative Analysis

arXiv: https://arxiv.org/abs/1701.02877 · HTML: https://mlsanity.com/paper/1701.02877/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jan 2017.

## Built on by (most cited first, 25 of 25)

- [Named Entity Recognition and Classification on Historical Documents: A Survey](https://mlsanity.com/paper/2109.11406/) (2021)
- [Memorisation versus Generalisation in Pre-trained Language Models](https://mlsanity.com/paper/2105.00828/) (2021)
- [Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition](https://mlsanity.com/paper/2107.11610/) (2021)
- [MultiCoNER: A Large-scale Multilingual dataset for Complex Named Entity Recognition](https://mlsanity.com/paper/2208.14536/) (2022)
- [GENTLE: A Genre-Diverse Multilayer Challenge Set for English NLP and Linguistic Evaluation](https://mlsanity.com/paper/2306.01966/) (2023)
- [Analysing the Noise Model Error for Realistic Noisy Label Data](https://mlsanity.com/paper/2101.09763/) (2021)
- [What do we Really Know about State of the Art NER?](https://mlsanity.com/paper/2205.00034/) (2022)
- [Does Typological Blinding Impede Cross-Lingual Sharing?](https://mlsanity.com/paper/2101.11888/) (2021)
- [Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?](https://mlsanity.com/paper/2212.09747/) (2022)
- [ACLM: A Selective-Denoising based Generative Data Augmentation Approach for Low-Resource Complex NER](https://mlsanity.com/paper/2306.00928/) (2023)
- [How Do Your Biomedical Named Entity Recognition Models Generalize to Novel Entities?](https://mlsanity.com/paper/2101.00160/) (2021)
- [Partially Supervised Named Entity Recognition via the Expected Entity Ratio Loss](https://mlsanity.com/paper/2108.07216/) (2021)
- [On Evaluation of Document Classification using RVL-CDIP](https://mlsanity.com/paper/2306.12550/) (2023)
- [Information Redundancy and Biases in Public Document Information Extraction Benchmarks](https://mlsanity.com/paper/2304.14936/) (2023)
- [Separating Retention from Extraction in the Evaluation of End-to-end Relation Extraction](https://mlsanity.com/paper/2109.12008/) (2021)
- [UM6P-CS at SemEval-2022 Task 11: Enhancing Multilingual and Code-Mixed Complex Named Entity Recognition via Pseudo Labels using Multilingual Transformer](https://mlsanity.com/paper/2204.13515/) (2022)
- [LMN at SemEval-2022 Task 11: A Transformer-based System for English Named Entity Recognition](https://mlsanity.com/paper/2203.03546/) (2022)
- [IXA/Cogcomp at SemEval-2023 Task 2: Context-enriched Multilingual Named Entity Recognition using Knowledge Bases](https://mlsanity.com/paper/2304.10637/) (2023)
- [Do Language Models Learn about Legal Entity Types during Pretraining?](https://mlsanity.com/paper/2310.13092/) (2023)
- [Implications of Annotation Artifacts in Edge Probing Test Datasets](https://mlsanity.com/paper/2310.13856/) (2023)
- [TEAM-Atreides at SemEval-2022 Task 11: On leveraging data augmentation and ensemble to recognize complex Named Entities in Bangla](https://mlsanity.com/paper/2204.09964/) (2022)
- [SFE-AI at SemEval-2022 Task 11: Low-Resource Named Entity Recognition using Large Pre-trained Language Models](https://mlsanity.com/paper/2205.14660/) (2022)
- [Dynamic Named Entity Recognition](https://mlsanity.com/paper/2302.10314/) (2023)
- [A Corpus for Named Entity Recognition in Chinese Novels with Multi-genres](https://mlsanity.com/paper/2311.15509/) (2023)
- [On-Device Named-Entity Recognition: A Deployability Study of Accuracy, Cost, Reliability, and Confidence](https://mlsanity.com/paper/2610.00007/) (2026)

## Abstract (arXiv)

Named Entity Recognition (NER) is a key NLP task, which is all the more challenging on Web and user-generated content with their diverse and continuously changing language. This paper aims to quantify how this diversity impacts state-of-the-art NER methods, by measuring named entity (NE) and context variability, feature sparsity, and their effects on precision and recall. In particular, our findings indicate that NER approaches struggle to generalise in diverse genres with limited training data. Unseen NEs, in particular, play an important role, which have a higher incidence in diverse genres such as social media than in more regular genres such as newswire. Coupled with a higher incidence of unseen features more generally and the lack of large training corpora, this leads to significantly lower F1 scores for diverse genres as compared to more regular ones. We also find that leading systems rely heavily on surface forms found in training data, having problems generalising beyond these, and offer explanations for this observation.