# Evaluating the Crowd with Confidence

arXiv: https://arxiv.org/abs/1411.6562 · HTML: https://mlsanity.com/paper/1411.6562/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2014.

## Built on by (most cited first, 4 of 4)

- [Universalizing Weak Supervision](https://mlsanity.com/paper/2112.03865/) (2021)
- [Toward Annotator Group Bias in Crowdsourcing](https://mlsanity.com/paper/2110.08038/) (2021)
- [Comparing the Value of Labeled and Unlabeled Data in Method-of-Moments Latent Variable Estimation](https://mlsanity.com/paper/2103.02761/) (2021)
- [ALIVE: Warnings Before Exclusion in Budgeted Multi-Source Learning](https://mlsanity.com/paper/2607.29400/) (2026)

## Abstract (arXiv)

Worker quality control is a crucial aspect of crowdsourcing systems; typically occupying a large fraction of the time and money invested on crowdsourcing. In this work, we devise techniques to generate confidence intervals for worker error rate estimates, thereby enabling a better evaluation of worker quality. We show that our techniques generate correct confidence intervals on a range of real-world datasets, and demonstrate wide applicability by using them to evict poorly performing workers, and provide confidence intervals on the accuracy of the answers.