# Annotation Errors and NER: A Study with OntoNotes 5.0

arXiv: https://arxiv.org/abs/2406.19172 · HTML: https://mlsanity.com/paper/2406.19172/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 25 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 4 of 4)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [CrossWeigh: Training Named Entity Tagger from Imperfect Annotations](https://mlsanity.com/paper/1909.01441/) (2019)
- [Named Entity Recognition -- Is there a glass ceiling?](https://mlsanity.com/paper/1910.02403/) (2019)

## Built on by (most cited first, 4 of 4)

- [Test Set Quality in Multilingual LLM Evaluation](https://mlsanity.com/paper/2508.02635/) (2025)
- [EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI](https://mlsanity.com/paper/2511.08949/) (2025)
- [DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition](https://mlsanity.com/paper/2604.15866/) (2026)
- [A Scalable Framework for Automated NER Annotation Correction in Low-Resource Languages](https://mlsanity.com/paper/2609.18739/) (2026)

## Abstract (arXiv)

Named Entity Recognition (NER) is a well-studied problem in NLP. However, there is much less focus on studying NER datasets, compared to developing new NER models. In this paper, we employed three simple techniques to detect annotation errors in the OntoNotes 5.0 corpus for English NER, which is the largest available NER corpus for English. Our techniques corrected ~10% of the sentences in train/dev/test data. In terms of entity mentions, we corrected the span and/or type of ~8% of mentions in the dataset, while adding/deleting/splitting/merging a few more. These are large numbers of changes, considering the size of OntoNotes. We used three NER libraries to train, evaluate and compare the models trained with the original and the re-annotated datasets, which showed an average improvement of 1.23% in overall F-scores, with large (>10%) improvements for some of the entity types. While our annotation error detection methods are not exhaustive and there is some manual annotation effort involved, they are largely language agnostic and can be employed with other NER datasets, and other sequence labelling tasks.