# Incentivizing High Quality Crowdwork

arXiv: https://arxiv.org/abs/1503.05897 · HTML: https://mlsanity.com/paper/1503.05897/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2015.

## Built on by (most cited first, 12 of 12)

- [Analyzing Dataset Annotation Quality Management in the Wild](https://mlsanity.com/paper/2307.08153/) (2023)
- [What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?](https://mlsanity.com/paper/2106.00794/) (2021)
- [MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domain](https://mlsanity.com/paper/2106.01491/) (2021)
- [GAM Coach: Towards Interactive and User-centered Algorithmic Recourse](https://mlsanity.com/paper/2302.14165/) (2023)
- [Are Bounded Contracts Learnable and Approximately Optimal?](https://mlsanity.com/paper/2402.14486/) (2024)
- [Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection](https://mlsanity.com/paper/2308.12885/) (2023)
- [SelectMix: Enhancing Label Noise Robustness through Targeted Sample Mixing](https://mlsanity.com/paper/2509.11265/) (2025)
- [When Incentives Backfire, Data Stops Being Human](https://mlsanity.com/paper/2502.07732/) (2025)
- [Bias in the Loop: How Humans Evaluate AI-Generated Suggestions](https://mlsanity.com/paper/2509.08514/) (2025)
- [Trading-Off Payments and Accuracy in Online Classification with Paid Stochastic Experts](https://mlsanity.com/paper/2307.00836/) (2023)
- [Goal Recognition Design for General Behavioral Agents using Machine Learning](https://mlsanity.com/paper/2404.03054/) (2024)
- [Adaptive Crowdsourcing Via Self-Supervised Learning](https://mlsanity.com/paper/2401.13239/) (2024)

## Abstract (arXiv)

We study the causal effects of financial incentives on the quality of crowdwork. We focus on performance-based payments (PBPs), bonus payments awarded to workers for producing high quality work. We design and run randomized behavioral experiments on the popular crowdsourcing platform Amazon Mechanical Turk with the goal of understanding when, where, and why PBPs help, identifying properties of the payment, payment structure, and the task itself that make them most effective. We provide examples of tasks for which PBPs do improve quality. For such tasks, the effectiveness of PBPs is not too sensitive to the threshold for quality required to receive the bonus, while the magnitude of the bonus must be large enough to make the reward salient. We also present examples of tasks for which PBPs do not improve quality. Our results suggest that for PBPs to improve quality, the task must be effort-responsive: the task must allow workers to produce higher quality work by exerting more effort. We also give a simple method to determine if a task is effort-responsive a priori. Furthermore, our experiments suggest that all payments on Mechanical Turk are, to some degree, implicitly performance-based in that workers believe their work may be rejected if their performance is sufficiently poor. Finally, we propose a new model of worker behavior that extends the standard principal-agent model from economics to include a worker's subjective beliefs about his likelihood of being paid, and show that the predictions of this model are in line with our experimental findings. This model may be useful as a foundation for theoretical studies of incentives in crowdsourcing markets.