# Rethinking Negative Sampling for Handling Missing Entity Annotations

arXiv: https://arxiv.org/abs/2108.11607 · HTML: https://mlsanity.com/paper/2108.11607/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 26 Aug 2021.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [A Unified MRC Framework for Named Entity Recognition](https://mlsanity.com/paper/1910.11476/) (2019)
- [Named Entity Recognition as Dependency Parsing](https://mlsanity.com/paper/2005.07150/) (2020)
- [Semi-Supervised Sequence Modeling with Cross-View Training](https://mlsanity.com/paper/1809.08370/) (2018)
- [Automated Phrase Mining from Massive Text Corpora](https://mlsanity.com/paper/1702.04457/) (2017)
- [Learning Named Entity Tagger using Domain-Specific Dictionary](https://mlsanity.com/paper/1809.03599/) (2018)
- [Distantly Supervised Named Entity Recognition using Positive-Unlabeled Learning](https://mlsanity.com/paper/1906.01378/) (2019)
- [Empirical Analysis of Unlabeled Entity Problem in Named Entity Recognition](https://mlsanity.com/paper/2012.05426/) (2020)
- [Hierarchical Contextualized Representation for Named Entity Recognition](https://mlsanity.com/paper/1911.02257/) (2019)
- [SwellShark: A Generative Model for Biomedical Named Entity Recognition without Labeled Data](https://mlsanity.com/paper/1704.06360/) (2017)
- [Named Entity Recognition with Partially Annotated Training Data](https://mlsanity.com/paper/1909.09270/) (2019)
- [TexSmart: A Text Understanding System for Fine-Grained NER and Enhanced Semantic Analysis](https://mlsanity.com/paper/2012.15639/) (2020)
- [Segmenting Natural Language Sentences via Lexical Unit Analysis](https://mlsanity.com/paper/2012.05418/) (2020)

## Built on by (most cited first, 12 of 12)

- [Uncertainty in Natural Language Processing: Sources, Quantification, and Applications](https://mlsanity.com/paper/2306.04459/) (2023)
- [Does Negative Sampling Matter? A Review with Insights into its Theory and Applications](https://mlsanity.com/paper/2402.17238/) (2024)
- [Rethinking Negative Instances for Generative Named Entity Recognition](https://mlsanity.com/paper/2402.16602/) (2024)
- [SCL-RAI: Span-based Contrastive Learning with Retrieval Augmented Inference for Unlabeled Entity Problem in NER](https://mlsanity.com/paper/2209.01646/) (2022)
- [Learning "O" Helps for Learning More: Handling the Concealed Entity Problem for Class-incremental NER](https://mlsanity.com/paper/2210.04676/) (2022)
- [Towards Integration of Discriminability and Robustness for Document-Level Relation Extraction](https://mlsanity.com/paper/2304.00824/) (2023)
- [SANTA: Separate Strategies for Inaccurate and Incomplete Annotation Noise in Distantly-Supervised Named Entity Recognition](https://mlsanity.com/paper/2305.04076/) (2023)
- [MProto: Multi-Prototype Network with Denoised Optimal Transport for Distantly Supervised Named Entity Recognition](https://mlsanity.com/paper/2310.08298/) (2023)
- [Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Training](https://mlsanity.com/paper/2305.12634/) (2023)
- [Better Sampling of Negatives for Distantly Supervised Named Entity Recognition](https://mlsanity.com/paper/2305.13142/) (2023)
- [DynClean: Training Dynamics-based Label Cleaning for Distantly-Supervised Named Entity Recognition](https://mlsanity.com/paper/2504.04616/) (2025)
- [Towards DS-NER: Unveiling and Addressing Latent Noise in Distant Annotations](https://mlsanity.com/paper/2505.12454/) (2025)

## Abstract (arXiv)

Negative sampling is highly effective in handling missing annotations for named entity recognition (NER). One of our contributions is an analysis on how it makes sense through introducing two insightful concepts: missampling and uncertainty. Empirical studies show low missampling rate and high uncertainty are both essential for achieving promising performances with negative sampling. Based on the sparsity of named entities, we also theoretically derive a lower bound for the probability of zero missampling rate, which is only relevant to sentence length. The other contribution is an adaptive and weighted sampling distribution that further improves negative sampling via our former analysis. Experiments on synthetic datasets and well-annotated datasets (e.g., CoNLL-2003) show that our proposed approach benefits negative sampling in terms of F1 score and loss convergence. Besides, models with improved negative sampling have achieved new state-of-the-art results on real-world datasets (e.g., EC).