# Cheaper and Better: Selecting Good Workers for Crowdsourcing

arXiv: https://arxiv.org/abs/1502.00725 · HTML: https://mlsanity.com/paper/1502.00725/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Feb 2015.

## Built on by (most cited first, 4 of 4)

- [Towards Unbiased and Accurate Deferral to Multiple Experts](https://mlsanity.com/paper/2102.13004/) (2021)
- [Cross-domain-aware Worker Selection with Training for Crowdsourced Annotation](https://mlsanity.com/paper/2406.06977/) (2024)
- [Pre-trained Language Models as Re-Annotators](https://mlsanity.com/paper/2205.05368/) (2022)
- [Operational Collective Intelligence of Humans and Machines](https://mlsanity.com/paper/2402.13273/) (2024)

## Abstract (arXiv)

Crowdsourcing provides a popular paradigm for data collection at scale. We study the problem of selecting subsets of workers from a given worker pool to maximize the accuracy under a budget constraint. One natural question is whether we should hire as many workers as the budget allows, or restrict on a small number of top-quality workers. By theoretically analyzing the error rate of a typical setting in crowdsourcing, we frame the worker selection problem into a combinatorial optimization problem and propose an algorithm to solve it efficiently. Empirical results on both simulated and real-world datasets show that our algorithm is able to select a small number of high-quality workers, and performs as good as, sometimes even better than, the much larger crowds as the budget allows.