# CleanCoNLL: A Nearly Noise-Free Named Entity Recognition Dataset

arXiv: https://arxiv.org/abs/2310.16225 · HTML: https://mlsanity.com/paper/2310.16225/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 13 of 13)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Named Entity Recognition as Dependency Parsing](https://mlsanity.com/paper/2005.07150/) (2020)
- [Automated Concatenation of Embeddings for Structured Prediction](https://mlsanity.com/paper/2010.05006/) (2020)
- [CrossWeigh: Training Named Entity Tagger from Imperfect Annotations](https://mlsanity.com/paper/1909.01441/) (2019)
- [FLERT: Document-Level Features for Named Entity Recognition](https://mlsanity.com/paper/2011.06993/) (2020)
- [Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study](https://mlsanity.com/paper/2001.03844/) (2020)
- [Learning from Noisy Labels for Entity-Centric Information Extraction](https://mlsanity.com/paper/2104.08656/) (2021)
- [Detecting Label Errors in Token Classification Data](https://mlsanity.com/paper/2210.03920/) (2022)
- [Influence Functions for Sequence Tagging Models](https://mlsanity.com/paper/2210.14177/) (2022)
- [Learning from Noisy Crowd Labels with Logics](https://mlsanity.com/paper/2302.06337/) (2023)

## Built on by (most cited first, 11 of 11)

- [VariErr NLI: Separating Annotation Error from Human Label Variation](https://mlsanity.com/paper/2403.01931/) (2024)
- [From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered](https://mlsanity.com/paper/2506.07461/) (2025)
- [Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations](https://mlsanity.com/paper/2402.01423/) (2024)
- [CLEANANERCorp: Identifying and Correcting Incorrect Labels in the ANERcorp Dataset](https://mlsanity.com/paper/2408.12362/) (2024)
- [Hierarchical Text Classification with LLM-Refined Taxonomies](https://mlsanity.com/paper/2601.18375/) (2026)
- [NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition](https://mlsanity.com/paper/2405.07609/) (2024)
- [Semantically Orthogonal Framework for Citation Classification: Disentangling Intent and Content](https://mlsanity.com/paper/2601.05103/) (2026)
- [OntoKG: Ontology-Oriented Knowledge Graph Construction with Intrinsic-Relational Routing](https://mlsanity.com/paper/2604.02618/) (2026)
- [Revising RVL-CDIP: Quantifying Errors and Test-Train Overlap](https://mlsanity.com/paper/2606.31446/) (2026)
- [Error-Type-Aware Loss Reweighting for Robust Named Entity Recognition with Noisy LLM Labels](https://mlsanity.com/paper/2608.30827/) (2026)
- [A Scalable Framework for Automated NER Annotation Correction in Low-Resource Languages](https://mlsanity.com/paper/2609.18739/) (2026)

## Abstract (arXiv)

The CoNLL-03 corpus is arguably the most well-known and utilized benchmark dataset for named entity recognition (NER). However, prior works found significant numbers of annotation errors, incompleteness, and inconsistencies in the data. This poses challenges to objectively comparing NER approaches and analyzing their errors, as current state-of-the-art models achieve F1-scores that are comparable to or even exceed the estimated noise level in CoNLL-03. To address this issue, we present a comprehensive relabeling effort assisted by automatic consistency checking that corrects 7.0% of all labels in the English CoNLL-03. Our effort adds a layer of entity linking annotation both for better explainability of NER labels and as additional safeguard of annotation quality. Our experimental evaluation finds not only that state-of-the-art approaches reach significantly higher F1-scores (97.1%) on our data, but crucially that the share of correct predictions falsely counted as errors due to annotation noise drops from 47% to 6%. This indicates that our resource is well suited to analyze the remaining errors made by state-of-the-art models, and that the theoretical upper bound even on high resource, coarse-grained NER is not yet reached. To facilitate such analysis, we make CleanCoNLL publicly available to the research community.