# NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition

arXiv: https://arxiv.org/abs/2405.07609 · HTML: https://mlsanity.com/paper/2405.07609/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 13 May 2024.

## Builds on (most cited first, 21 of 21)

- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction](https://mlsanity.com/paper/2304.08085/) (2023)
- [Deep Learning is Robust to Massive Label Noise](https://mlsanity.com/paper/1705.10694/) (2017)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision](https://mlsanity.com/paper/2006.15509/) (2020)
- [GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer](https://mlsanity.com/paper/2311.08526/) (2023)
- [Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach](https://mlsanity.com/paper/2010.07835/) (2020)
- [CrossWeigh: Training Named Entity Tagger from Imperfect Annotations](https://mlsanity.com/paper/1909.01441/) (2019)
- [ProMix: Combating Label Noise via Maximizing Clean Sample Utility](https://mlsanity.com/paper/2207.10276/) (2022)
- [Named Entity Recognition without Labelled Data: A Weak Supervision Approach](https://mlsanity.com/paper/2004.14723/) (2020)
- [FLERT: Document-Level Features for Named Entity Recognition](https://mlsanity.com/paper/2011.06993/) (2020)
- [Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future](https://mlsanity.com/paper/2206.02280/) (2022)
- [Learning from Noisy Labels for Entity-Centric Information Extraction](https://mlsanity.com/paper/2104.08656/) (2021)
- [Named Entity Recognition with Partially Annotated Training Data](https://mlsanity.com/paper/1909.09270/) (2019)
- [Is BERT Robust to Label Noise? A Study on Learning with Noisy Labels in Text Classification](https://mlsanity.com/paper/2204.09371/) (2022)
- [Analysing the Noise Model Error for Realistic Noisy Label Data](https://mlsanity.com/paper/2101.09763/) (2021)
- [CleanCoNLL: A Nearly Noise-Free Named Entity Recognition Dataset](https://mlsanity.com/paper/2310.16225/) (2023)
- [NoisywikiHow: A Benchmark for Learning with Real-world Noisy Labels in Natural Language Processing](https://mlsanity.com/paper/2305.10709/) (2023)
- [Fabricator: An Open Source Toolkit for Generating Labeled Training Data with Teacher LLMs](https://mlsanity.com/paper/2309.09582/) (2023)

## Built on by (most cited first, 1 of 1)

- [Label Convergence: Defining an Upper Performance Bound in Object Recognition through Contradictory Annotations](https://mlsanity.com/paper/2409.09412/) (2024)

## Abstract (arXiv)

Available training data for named entity recognition (NER) often contains a significant percentage of incorrect labels for entity types and entity boundaries. Such label noise poses challenges for supervised learning and may significantly deteriorate model quality. To address this, prior work proposed various noise-robust learning approaches capable of learning from data with partially incorrect labels. These approaches are typically evaluated using simulated noise where the labels in a clean dataset are automatically corrupted. However, as we show in this paper, this leads to unrealistic noise that is far easier to handle than real noise caused by human error or semi-automatic annotation. To enable the study of the impact of various types of real noise, we introduce NoiseBench, an NER benchmark consisting of clean training data corrupted with 6 types of real noise, including expert errors, crowdsourcing errors, automatic annotation errors and LLM errors. We present an analysis that shows that real noise is significantly more challenging than simulated noise, and show that current state-of-the-art models for noise-robust learning fall far short of their theoretically achievable upper bound. We release NoiseBench to the research community.