# Empirical Analysis of Unlabeled Entity Problem in Named Entity Recognition

arXiv: https://arxiv.org/abs/2012.05426 · HTML: https://mlsanity.com/paper/2012.05426/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2020.

## Built on by (most cited first, 30 of 35)

- [GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer](https://mlsanity.com/paper/2311.08526/) (2023)
- [Revisiting DocRED -- Addressing the False Negative Problem in Relation Extraction](https://mlsanity.com/paper/2205.12696/) (2022)
- [GL-GIN: Fast and Accurate Non-Autoregressive Model for Joint Multiple Intent Detection and Slot Filling](https://mlsanity.com/paper/2106.01925/) (2021)
- [Searching for Optimal Subword Tokenization in Cross-domain NER](https://mlsanity.com/paper/2206.03352/) (2022)
- [Does Negative Sampling Matter? A Review with Insights into its Theory and Applications](https://mlsanity.com/paper/2402.17238/) (2024)
- [Distantly-Supervised Named Entity Recognition with Adaptive Teacher Learning and Fine-grained Student Ensemble](https://mlsanity.com/paper/2212.06522/) (2022)
- [Rethinking Negative Instances for Generative Named Entity Recognition](https://mlsanity.com/paper/2402.16602/) (2024)
- [Rethinking Negative Sampling for Handling Missing Entity Annotations](https://mlsanity.com/paper/2108.11607/) (2021)
- [Improving Distantly-Supervised Named Entity Recognition with Self-Collaborative Denoising Learning](https://mlsanity.com/paper/2110.04429/) (2021)
- [SCL-RAI: Span-based Contrastive Learning with Retrieval Augmented Inference for Unlabeled Entity Problem in NER](https://mlsanity.com/paper/2209.01646/) (2022)
- [GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation](https://mlsanity.com/paper/2211.10330/) (2022)
- [What do we Really Know about State of the Art NER?](https://mlsanity.com/paper/2205.00034/) (2022)
- [Learning "O" Helps for Learning More: Handling the Concealed Entity Problem for Class-incremental NER](https://mlsanity.com/paper/2210.04676/) (2022)
- [Partially Supervised Named Entity Recognition via the Expected Entity Ratio Loss](https://mlsanity.com/paper/2108.07216/) (2021)
- [A Unified One-Step Solution for Aspect Sentiment Quad Prediction](https://mlsanity.com/paper/2306.04152/) (2023)
- [Deep Span Representations for Named Entity Recognition](https://mlsanity.com/paper/2210.04182/) (2022)
- [Towards Integration of Discriminability and Robustness for Document-Level Relation Extraction](https://mlsanity.com/paper/2304.00824/) (2023)
- [SANTA: Separate Strategies for Inaccurate and Incomplete Annotation Noise in Distantly-Supervised Named Entity Recognition](https://mlsanity.com/paper/2305.04076/) (2023)
- [MProto: Multi-Prototype Network with Denoised Optimal Transport for Distantly Supervised Named Entity Recognition](https://mlsanity.com/paper/2310.08298/) (2023)
- [Online Coreference Resolution for Dialogue Processing: Improving Mention-Linking on Real-Time Conversations](https://mlsanity.com/paper/2205.10670/) (2022)
- [Recognizing Nested Entities from Flat Supervision: A New NER Subtask, Feasibility and Challenges](https://mlsanity.com/paper/2211.00301/) (2022)
- [Partial Annotation Learning for Biomedical Entity Recognition](https://mlsanity.com/paper/2305.13120/) (2023)
- [Better Sampling of Negatives for Distantly Supervised Named Entity Recognition](https://mlsanity.com/paper/2305.13142/) (2023)
- [Class-Adaptive Self-Training for Relation Extraction with Incompletely Annotated Training Data](https://mlsanity.com/paper/2306.09697/) (2023)
- [Improving the Robustness of Distantly-Supervised Named Entity Recognition via Uncertainty-Aware Teacher Learning and Student-Student Collaborative Learning](https://mlsanity.com/paper/2311.08010/) (2023)
- [Span-Oriented Information Extraction -- A Unifying Perspective on Information Extraction](https://mlsanity.com/paper/2403.15453/) (2024)
- [Combining Supervised Learning and Reinforcement Learning for Multi-Label Classification Tasks with Partial Labels](https://mlsanity.com/paper/2406.16293/) (2024)
- [DynClean: Training Dynamics-based Label Cleaning for Distantly-Supervised Named Entity Recognition](https://mlsanity.com/paper/2504.04616/) (2025)
- [Neural Sequence Segmentation as Determining the Leftmost Segments](https://mlsanity.com/paper/2104.07217/) (2021)
- [Filtered Semi-Markov CRF](https://mlsanity.com/paper/2311.18028/) (2023)

## Abstract (arXiv)

In many scenarios, named entity recognition (NER) models severely suffer from unlabeled entity problem, where the entities of a sentence may not be fully annotated. Through empirical studies performed on synthetic datasets, we find two causes of performance degradation. One is the reduction of annotated entities and the other is treating unlabeled entities as negative instances. The first cause has less impact than the second one and can be mitigated by adopting pretraining language models. The second cause seriously misguides a model in training and greatly affects its performances. Based on the above observations, we propose a general approach, which can almost eliminate the misguidance brought by unlabeled entities. The key idea is to use negative sampling that, to a large extent, avoids training NER models with unlabeled entities. Experiments on synthetic datasets and real-world datasets show that our model is robust to unlabeled entity problem and surpasses prior baselines. On well-annotated datasets, our model is competitive with the state-of-the-art method.