# Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution

arXiv: https://arxiv.org/abs/2012.04169 · HTML: https://mlsanity.com/paper/2012.04169/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Dec 2020.

## Built on by (most cited first, 4 of 4)

- [What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?](https://mlsanity.com/paper/2106.00794/) (2021)
- [Toward More Effective Human Evaluation for Machine Translation](https://mlsanity.com/paper/2204.05307/) (2022)
- [Leveraging Human Feedback to Scale Educational Datasets: Combining Crowdworkers and Comparative Judgement](https://mlsanity.com/paper/2305.12894/) (2023)
- [Dynamic Human Evaluation for Relative Model Comparisons](https://mlsanity.com/paper/2112.08048/) (2021)

## Abstract (arXiv)

This paper develops and implements a scalable methodology for (a) estimating the noisiness of labels produced by a typical crowdsourcing semantic annotation task, and (b) reducing the resulting error of the labeling process by as much as 20-30% in comparison to other common labeling strategies. Importantly, this new approach to the labeling process, which we name Dynamic Automatic Conflict Resolution (DACR), does not require a ground truth dataset and is instead based on inter-project annotation inconsistencies. This makes DACR not only more accurate but also available to a broad range of labeling tasks. In what follows we present results from a text classification task performed at scale for a commercial personal assistant, and evaluate the inherent ambiguity uncovered by this annotation strategy as compared to other common labeling strategies.