# Noisy-Labeled NER with Confidence Estimation

arXiv: https://arxiv.org/abs/2104.04318 · HTML: https://mlsanity.com/paper/2104.04318/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 9 Apr 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision](https://mlsanity.com/paper/2006.15509/) (2020)
- [Learning Named Entity Tagger using Domain-Specific Dictionary](https://mlsanity.com/paper/1809.03599/) (2018)
- [Distantly Supervised Named Entity Recognition using Positive-Unlabeled Learning](https://mlsanity.com/paper/1906.01378/) (2019)
- [Low-Resource Name Tagging Learned with Weakly Labeled Data](https://mlsanity.com/paper/1908.09659/) (2019)
- [Named Entity Recognition with Partially Annotated Training Data](https://mlsanity.com/paper/1909.09270/) (2019)
- [Training a Neural Network in a Low-Resource Setting on Automatically Annotated Noisy Data](https://mlsanity.com/paper/1807.00745/) (2018)

## Built on by (most cited first, 28 of 28)

- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [Generative Knowledge Graph Construction: A Review](https://mlsanity.com/paper/2210.12714/) (2022)
- [Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion](https://mlsanity.com/paper/2205.02357/) (2022)
- [Gradient Imitation Reinforcement Learning for Low Resource Relation Extraction](https://mlsanity.com/paper/2109.06415/) (2021)
- [InstructionNER: A Multi-Task Instruction-Based Generative Framework for Few-shot NER](https://mlsanity.com/paper/2203.03903/) (2022)
- [LightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Prompting](https://mlsanity.com/paper/2109.00720/) (2021)
- [CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoning](https://mlsanity.com/paper/2305.04808/) (2023)
- [Ontology-enhanced Prompt-tuning for Few-shot Learning](https://mlsanity.com/paper/2201.11332/) (2022)
- [Optimizing Bi-Encoder for Named Entity Recognition via Contrastive Learning](https://mlsanity.com/paper/2208.14565/) (2022)
- [Distantly Supervised Named Entity Recognition via Confidence-Based Multi-Class Positive and Unlabeled Learning](https://mlsanity.com/paper/2204.09589/) (2022)
- [SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents](https://mlsanity.com/paper/2410.21155/) (2024)
- [DeepKE: A Deep Learning Based Knowledge Extraction Toolkit for Knowledge Base Population](https://mlsanity.com/paper/2201.03335/) (2022)
- [One Model for All Domains: Collaborative Domain-Prefix Tuning for Cross-Domain NER](https://mlsanity.com/paper/2301.10410/) (2023)
- [Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding](https://mlsanity.com/paper/2104.06393/) (2021)
- [Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction](https://mlsanity.com/paper/2205.03521/) (2022)
- [Simple Questions Generate Named Entity Recognition Datasets](https://mlsanity.com/paper/2112.08808/) (2021)
- [Towards Realistic Low-resource Relation Extraction: A Benchmark with Empirical Baseline Study](https://mlsanity.com/paper/2210.10678/) (2022)
- [Re-Examine Distantly Supervised NER: A New Benchmark and a Simple Approach](https://mlsanity.com/paper/2402.14948/) (2024)
- [Schema-adaptable Knowledge Graph Construction](https://mlsanity.com/paper/2305.08703/) (2023)
- [Integrating Large Pre-trained Models into Multimodal Named Entity Recognition with Evidential Fusion](https://mlsanity.com/paper/2306.16991/) (2023)
- [Improve Event Extraction via Self-Training with Gradient Guidance](https://mlsanity.com/paper/2205.12490/) (2022)
- [Better Sampling of Negatives for Distantly Supervised Named Entity Recognition](https://mlsanity.com/paper/2305.13142/) (2023)
- [Out-of-Distribution Generalization in Text Classification: Past, Present, and Future](https://mlsanity.com/paper/2305.14104/) (2023)
- [Few Dimensions are Enough: Fine-tuning BERT with Selected Dimensions Revealed Its Redundant Nature](https://mlsanity.com/paper/2504.04966/) (2025)
- [Gradient Imitation Reinforcement Learning for General Low-Resource Information Extraction](https://mlsanity.com/paper/2211.06014/) (2022)
- [A Curriculum View of Robust Loss Functions](https://mlsanity.com/paper/2305.02139/) (2023)
- [Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instances](https://mlsanity.com/paper/2310.16790/) (2023)
- [Error-Type-Aware Loss Reweighting for Robust Named Entity Recognition with Noisy LLM Labels](https://mlsanity.com/paper/2608.30827/) (2026)

## Abstract (arXiv)

Recent studies in deep learning have shown significant progress in named entity recognition (NER). Most existing works assume clean data annotation, yet a fundamental challenge in real-world scenarios is the large amount of noise from a variety of sources (e.g., pseudo, weak, or distant annotations). This work studies NER under a noisy labeled setting with calibrated confidence estimation. Based on empirical observations of different training dynamics of noisy and clean labels, we propose strategies for estimating confidence scores based on local and global independence assumptions. We partially marginalize out labels of low confidence with a CRF model. We further propose a calibration method for confidence scores based on the structure of entity labels. We integrate our approach into a self-training framework for boosting performance. Experiments in general noisy settings with four languages and distantly labeled settings demonstrate the effectiveness of our method. Our code can be found at https://github.com/liukun95/Noisy-NER-Confidence-Estimation