# Analysis of Minimax Error Rate for Crowdsourcing and Its Application to Worker Clustering Model

arXiv: https://arxiv.org/abs/1802.04551 · HTML: https://mlsanity.com/paper/1802.04551/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2018.

## Built on by (most cited first, 6 of 6)

- [Personalized Language Modeling from Personalized Human Feedback](https://mlsanity.com/paper/2402.05133/) (2024)
- [Improve Learning from Crowds via Generative Augmentation](https://mlsanity.com/paper/2107.10449/) (2021)
- [Identify ambiguous tasks combining crowdsourced labels by weighting Areas Under the Margin](https://mlsanity.com/paper/2209.15380/) (2022)
- [Mixture of Experts based Multi-task Supervise Learning from Crowds](https://mlsanity.com/paper/2407.13268/) (2024)
- [A Symmetric Loss Perspective of Reliable Machine Learning](https://mlsanity.com/paper/2101.01366/) (2021)
- [Towards AI-Empowered Crowdsourcing](https://mlsanity.com/paper/2212.14676/) (2022)

## Abstract (arXiv)

While crowdsourcing has become an important means to label data, there is great interest in estimating the ground truth from unreliable labels produced by crowdworkers. The Dawid and Skene (DS) model is one of the most well-known models in the study of crowdsourcing. Despite its practical popularity, theoretical error analysis for the DS model has been conducted only under restrictive assumptions on class priors, confusion matrices, or the number of labels each worker provides. In this paper, we derive a minimax error rate under more practical setting for a broader class of crowdsourcing models including the DS model as a special case. We further propose the worker clustering model, which is more practical than the DS model under real crowdsourcing settings. The wide applicability of our theoretical analysis allows us to immediately investigate the behavior of this proposed model, which can not be analyzed by existing studies. Experimental results showed that there is a strong similarity between the lower bound of the minimax error rate derived by our theoretical analysis and the empirical error of the estimated value.