# Comprehensive and Reliable Crowd Assessment Algorithms

arXiv: https://arxiv.org/abs/1411.3377 · HTML: https://mlsanity.com/paper/1411.3377/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2014.

## Built on by (most cited first, 5 of 5)

- [Universalizing Weak Supervision](https://mlsanity.com/paper/2112.03865/) (2021)
- [Towards Artificial Intelligence Enabled Financial Crime Detection](https://mlsanity.com/paper/2105.10866/) (2021)
- [An Adaptive Method for Weak Supervision with Drifting Data](https://mlsanity.com/paper/2306.01658/) (2023)
- [Leveraging Large Language Models for Structure Learning in Prompted Weak Supervision](https://mlsanity.com/paper/2402.01867/) (2024)
- [AutoWS: Automated Weak Supervision Framework for Text Classification](https://mlsanity.com/paper/2302.03297/) (2023)

## Abstract (arXiv)

Evaluating workers is a critical aspect of any crowdsourcing system. In this paper, we devise techniques for evaluating workers by finding confidence intervals on their error rates. Unlike prior work, we focus on "conciseness"---that is, giving as tight a confidence interval as possible. Conciseness is of utmost importance because it allows us to be sure that we have the best guarantee possible on worker error rate. Also unlike prior work, we provide techniques that work under very general scenarios, such as when not all workers have attempted every task (a fairly common scenario in practice), when tasks have non-boolean responses, and when workers have different biases for positive and negative tasks. We demonstrate conciseness as well as accuracy of our confidence intervals by testing them on a variety of conditions and multiple real-world datasets.