# CrossWeigh: Training Named Entity Tagger from Imperfect Annotations

arXiv: https://arxiv.org/abs/1909.01441 · HTML: https://mlsanity.com/paper/1909.01441/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2019.

## Built on by (most cited first, 30 of 60)

- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://mlsanity.com/paper/2304.01852/) (2023)
- [Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning](https://mlsanity.com/paper/2105.03654/) (2021)
- [UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition](https://mlsanity.com/paper/2308.03279/) (2023)
- [Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future](https://mlsanity.com/paper/2206.02280/) (2022)
- [VariErr NLI: Separating Annotation Error from Human Label Variation](https://mlsanity.com/paper/2403.01931/) (2024)
- [Analyzing Dataset Annotation Quality Management in the Wild](https://mlsanity.com/paper/2307.08153/) (2023)
- [T-NER: An All-Round Python Library for Transformer-based Named Entity Recognition](https://mlsanity.com/paper/2209.12616/) (2022)
- [Learning from Noisy Labels for Entity-Centric Information Extraction](https://mlsanity.com/paper/2104.08656/) (2021)
- [Open, Closed, or Small Language Models for Text Classification?](https://mlsanity.com/paper/2308.10092/) (2023)
- [Memorisation versus Generalisation in Pre-trained Language Models](https://mlsanity.com/paper/2105.00828/) (2021)
- [Boundary Smoothing for Named Entity Recognition](https://mlsanity.com/paper/2204.12031/) (2022)
- [DAMO-NLP at SemEval-2022 Task 11: A Knowledge-based System for Multilingual Named Entity Recognition](https://mlsanity.com/paper/2203.00545/) (2022)
- [WeDef: Weakly Supervised Backdoor Defense for Text Classification](https://mlsanity.com/paper/2205.11803/) (2022)
- [Rethinking Negative Instances for Generative Named Entity Recognition](https://mlsanity.com/paper/2402.16602/) (2024)
- [Detecting Label Errors in Token Classification Data](https://mlsanity.com/paper/2210.03920/) (2022)
- [Incorporating Context into Subword Vocabularies](https://mlsanity.com/paper/2210.07095/) (2022)
- [CleanCoNLL: A Nearly Noise-Free Named Entity Recognition Dataset](https://mlsanity.com/paper/2310.16225/) (2023)
- [ChatEL: Entity Linking with Chatbots](https://mlsanity.com/paper/2402.14858/) (2024)
- [ToNER: Type-oriented Named Entity Recognition with Generative Language Model](https://mlsanity.com/paper/2404.09145/) (2024)
- [Validating Label Consistency in NER Data Annotation](https://mlsanity.com/paper/2101.08698/) (2021)
- [Knodle: Modular Weakly Supervised Learning with PyTorch](https://mlsanity.com/paper/2104.11557/) (2021)
- [RexUIE: A Recursive Method with Explicit Schema Instructor for Universal Information Extraction](https://mlsanity.com/paper/2304.14770/) (2023)
- [Pretraining without Wordpieces: Learning Over a Vocabulary of Millions of Words](https://mlsanity.com/paper/2202.12142/) (2022)
- [Informed Named Entity Recognition Decoding for Generative Language Models](https://mlsanity.com/paper/2308.07791/) (2023)
- [Beyond Isolation: Multi-Agent Synergy for Improving Knowledge Graph Construction](https://mlsanity.com/paper/2312.03022/) (2023)
- [Multilingual Prompt Engineering in Large Language Models: A Survey Across NLP Tasks](https://mlsanity.com/paper/2505.11665/) (2025)
- [Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations](https://mlsanity.com/paper/2402.01423/) (2024)
- [SCANNER: Knowledge-Enhanced Approach for Robust Multi-modal Named Entity Recognition of Unseen Entities](https://mlsanity.com/paper/2404.01914/) (2024)

## Abstract (arXiv)

Everyone makes mistakes. So do human annotators when curating labels for named entity recognition (NER). Such label mistakes might hurt model training and interfere model comparison. In this study, we dive deep into one of the widely-adopted NER benchmark datasets, CoNLL03 NER. We are able to identify label mistakes in about 5.38% test sentences, which is a significant ratio considering that the state-of-the-art test F1 score is already around 93%. Therefore, we manually correct these label mistakes and form a cleaner test set. Our re-evaluation of popular models on this corrected test set leads to more accurate assessments, compared to those on the original test set. More importantly, we propose a simple yet effective framework, CrossWeigh, to handle label mistakes during NER model training. Specifically, it partitions the training data into several folds and train independent NER models to identify potential mistakes in each fold. Then it adjusts the weights of training data accordingly to train the final NER model. Extensive experiments demonstrate significant improvements of plugging various NER models into our proposed framework on three datasets. All implementations and corrected test set are available at our Github repo: https://github.com/ZihanWangKi/CrossWeigh.