# Recovering Top-Two Answers and Confusion Probability in Multi-Choice Crowdsourcing

arXiv: https://arxiv.org/abs/2301.00006 · HTML: https://mlsanity.com/paper/2301.00006/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 47 references that are in the map.
- First posted on arXiv 29 Dec 2022.

## Builds on (most cited first, 18 of 18)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcing](https://mlsanity.com/paper/1406.3824/) (2014)
- [PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review](https://mlsanity.com/paper/1806.06237/) (2018)
- [Crowdsourcing via Pairwise Co-occurrences: Identifiability and Algorithms](https://mlsanity.com/paper/1909.12325/) (2019)
- [Error Rate Bounds and Iterative Weighted Majority Voting for Crowdsourcing](https://mlsanity.com/paper/1411.4086/) (2014)
- [Adversarial Crowdsourcing Through Robust Rank-One Matrix Completion](https://mlsanity.com/paper/2010.12181/) (2020)
- [A Permutation-based Model for Crowd Labeling: Optimal Estimation and Robustness](https://mlsanity.com/paper/1606.09632/) (2016)
- [Consistency of weighted majority votes](https://mlsanity.com/paper/1312.0451/) (2013)
- [Gradient Descent for Sparse Rank-One Matrix Completion for Crowd-Sourced Aggregation of Sparsely Interacting Workers](https://mlsanity.com/paper/1904.11608/) (2019)
- [Spectral Clustering via the Power Method -- Provably](https://mlsanity.com/paper/1311.2854/) (2013)
- [Exact Exponent in Optimal Rates for Crowdsourcing](https://mlsanity.com/paper/1605.07696/) (2016)
- [GenLabel: Mixup Relabeling using Generative Models](https://mlsanity.com/paper/2201.02354/) (2022)
- [Achieving Budget-optimality with Adaptive Schemes in Crowdsourcing](https://mlsanity.com/paper/1602.03481/) (2016)
- [Minimax Optimal Convergence Rates for Estimating Ground Truth from Crowdsourced Labels](https://mlsanity.com/paper/1310.5764/) (2013)
- [Reducing Crowdsourcing to Graphon Estimation, Statistically](https://mlsanity.com/paper/1703.08085/) (2017)
- [Truth Discovery in Sequence Labels from Crowds](https://mlsanity.com/paper/2109.04470/) (2021)
- [Integrating Rankings into Quantized Scores in Peer Review](https://mlsanity.com/paper/2204.03505/) (2022)

## Abstract (arXiv)

Crowdsourcing has emerged as an effective platform for labeling large amounts of data in a cost- and time-efficient manner. Most previous work has focused on designing an efficient algorithm to recover only the ground-truth labels of the data. In this paper, we consider multi-choice crowdsourcing tasks with the goal of recovering not only the ground truth, but also the most confusing answer and the confusion probability. The most confusing answer provides useful information about the task by revealing the most plausible answer other than the ground truth and how plausible it is. To theoretically analyze such scenarios, we propose a model in which there are the top two plausible answers for each task, distinguished from the rest of the choices. Task difficulty is quantified by the probability of confusion between the top two, and worker reliability is quantified by the probability of giving an answer among the top two. Under this model, we propose a two-stage inference algorithm to infer both the top two answers and the confusion probability. We show that our algorithm achieves the minimax optimal convergence rate. We conduct both synthetic and real data experiments and demonstrate that our algorithm outperforms other recent algorithms. We also show the applicability of our algorithms in inferring the difficulty of tasks and in training neural networks with top-two soft labels.