# Validating Label Consistency in NER Data Annotation

arXiv: https://arxiv.org/abs/2101.08698 · HTML: https://mlsanity.com/paper/2101.08698/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 16 references that are in the map.
- First posted on arXiv 21 Jan 2021.

## Builds on (most cited first, 6 of 6)

- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [A General Framework for Information Extraction using Dynamic Span Graphs](https://mlsanity.com/paper/1904.03296/) (2019)
- [CrossWeigh: Training Named Entity Tagger from Imperfect Annotations](https://mlsanity.com/paper/1909.01441/) (2019)
- [Empower Sequence Labeling with Task-Aware Neural Language Model](https://mlsanity.com/paper/1709.04109/) (2017)

## Built on by (most cited first, 7 of 7)

- [Large Language Models Can Self-Correct with Key Condition Verification](https://mlsanity.com/paper/2405.14092/) (2024)
- [ChatEL: Entity Linking with Chatbots](https://mlsanity.com/paper/2402.14858/) (2024)
- [The Problem of Coherence in Natural Language Explanations of Recommendations](https://mlsanity.com/paper/2312.11356/) (2023)
- [Statistical Dataset Evaluation: Reliability, Difficulty, and Validity](https://mlsanity.com/paper/2212.09272/) (2022)
- [A survey on bias in machine learning research](https://mlsanity.com/paper/2308.11254/) (2023)
- [Evolution of ESG-focused DLT Research: An NLP Analysis of the Literature](https://mlsanity.com/paper/2308.12420/) (2023)
- [Uncovering Misattributed Suicide Causes through Annotation Inconsistency Detection in Death Investigation Notes](https://mlsanity.com/paper/2403.19432/) (2024)

## Abstract (arXiv)

Data annotation plays a crucial role in ensuring your named entity recognition (NER) projects are trained with the right information to learn from. Producing the most accurate labels is a challenge due to the complexity involved with annotation. Label inconsistency between multiple subsets of data annotation (e.g., training set and test set, or multiple training subsets) is an indicator of label mistakes. In this work, we present an empirical method to explore the relationship between label (in-)consistency and NER model performance. It can be used to validate the label consistency (or catches the inconsistency) in multiple sets of NER data annotation. In experiments, our method identified the label inconsistency of test data in SCIERC and CoNLL03 datasets (with 26.7% and 5.4% label mistakes). It validated the consistency in the corrected version of both datasets.