# A Permutation-based Model for Crowd Labeling: Optimal Estimation and Robustness

arXiv: https://arxiv.org/abs/1606.09632 · HTML: https://mlsanity.com/paper/1606.09632/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jun 2016.

## Built on by (most cited first, 13 of 13)

- [CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription](https://mlsanity.com/paper/2107.01091/) (2021)
- [Active Ranking of Experts Based on their Performances in Many Tasks](https://mlsanity.com/paper/2306.02628/) (2023)
- [Optimal detection of the feature matching map in presence of noise and outliers](https://mlsanity.com/paper/2106.07044/) (2021)
- [A Worker-Task Specialization Model for Crowdsourcing: Efficient Inference and Fundamental Limits](https://mlsanity.com/paper/2111.12550/) (2021)
- [Scalable Interactive Machine Learning for Future Command and Control](https://mlsanity.com/paper/2402.06501/) (2024)
- [Matching Map Recovery with an Unknown Number of Outliers](https://mlsanity.com/paper/2210.13354/) (2022)
- [Modeling and Correcting Bias in Sequential Evaluation](https://mlsanity.com/paper/2205.01607/) (2022)
- [Recovering Top-Two Answers and Confusion Probability in Multi-Choice Crowdsourcing](https://mlsanity.com/paper/2301.00006/) (2022)
- [Spectral Clustering for Crowdsourcing with Inherently Distinct Task Types](https://mlsanity.com/paper/2302.07393/) (2023)
- [How to Shrink Confidence Sets for Many Equivalent Discrete Distributions?](https://mlsanity.com/paper/2407.15662/) (2024)
- [The Majority Vote Paradigm Shift: When Popular Meets Optimal](https://mlsanity.com/paper/2502.12581/) (2025)
- [Statistical and computational challenges in ranking](https://mlsanity.com/paper/2512.21111/) (2025)
- [Statistical and Structural Approaches to Algorithmic Fairness](https://mlsanity.com/paper/2606.26200/) (2026)

## Abstract (arXiv)

The task of aggregating and denoising crowd-labeled data has gained increased significance with the advent of crowdsourcing platforms and massive datasets. We propose a permutation-based model for crowd labeled data that is a significant generalization of the classical Dawid-Skene model, and introduce a new error metric by which to compare different estimators. We derive global minimax rates for the permutation-based model that are sharp up to logarithmic factors, and match the minimax lower bounds derived under the simpler Dawid-Skene model. We then design two computationally-efficient estimators: the WAN estimator for the setting where the ordering of workers in terms of their abilities is approximately known, and the OBI-WAN estimator where that is not known. For each of these estimators, we provide non-asymptotic bounds on their performance. We conduct synthetic simulations and experiments on real-world crowdsourcing data, and the experimental results corroborate our theoretical findings.