# Unsupervised Crowdsourcing with Accuracy and Cost Guarantees

arXiv: https://arxiv.org/abs/2207.01988 · HTML: https://mlsanity.com/paper/2207.01988/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 35 references that are in the map.
- First posted on arXiv 5 Jul 2022.

## Builds on (most cited first, 7 of 7)

- [On the Complexity of Best Arm Identification in Multi-Armed Bandit Models](https://mlsanity.com/paper/1407.4443/) (2014)
- [Optimal Best Arm Identification with Fixed Confidence](https://mlsanity.com/paper/1602.04589/) (2016)
- [Automatic Detection of Fake News](https://mlsanity.com/paper/1708.07104/) (2017)
- [Statistical guarantees for the EM algorithm: From population to sample-based analysis](https://mlsanity.com/paper/1408.2156/) (2014)
- [Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcing](https://mlsanity.com/paper/1406.3824/) (2014)
- [Achieving Budget-optimality with Adaptive Schemes in Crowdsourcing](https://mlsanity.com/paper/1602.03481/) (2016)
- [Reducing Crowdsourcing to Graphon Estimation, Statistically](https://mlsanity.com/paper/1703.08085/) (2017)

## Built on by (most cited first, 1 of 1)

- [FUSE: Ensembling Verifiers with Zero Labeled Data](https://mlsanity.com/paper/2604.18547/) (2026)

## Abstract (arXiv)

We consider the problem of cost-optimal utilization of a crowdsourcing platform for binary, unsupervised classification of a collection of items, given a prescribed error threshold. Workers on the crowdsourcing platform are assumed to be divided into multiple classes, based on their skill, experience, and/or past performance. We model each worker class via an unknown confusion matrix, and a (known) price to be paid per label prediction. For this setting, we propose algorithms for acquiring label predictions from workers, and for inferring the true labels of items. We prove that if the number of (unlabeled) items available is large enough, our algorithms satisfy the prescribed error thresholds, incurring a cost that is near-optimal. Finally, we validate our algorithms, and some heuristics inspired by them, through an extensive case study.