# Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions

arXiv: https://arxiv.org/abs/1801.02546 · HTML: https://mlsanity.com/paper/1801.02546/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jan 2018.

## Built on by (most cited first, 16 of 16)

- [If in a Crowdsourced Data Annotation Pipeline, a GPT-4](https://mlsanity.com/paper/2402.16795/) (2024)
- [Lego: Learning to Disentangle and Invert Personalized Concepts Beyond Object Appearance in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2311.13833/) (2023)
- [API-guided Dataset Synthesis to Finetune Large Code Models](https://mlsanity.com/paper/2408.08343/) (2024)
- [Few-shot LLM Synthetic Data with Distribution Matching](https://mlsanity.com/paper/2502.08661/) (2025)
- [Training Affective Computer Vision Models by Crowdsourcing Soft-Target Labels](https://mlsanity.com/paper/2101.03477/) (2021)
- [COPA-SSE: Semi-structured Explanations for Commonsense Reasoning](https://mlsanity.com/paper/2201.06777/) (2022)
- [In the Picture: Medical Imaging Datasets, Artifacts, and their Living Review](https://mlsanity.com/paper/2501.10727/) (2025)
- [ProVe: A Pipeline for Automated Provenance Verification of Knowledge Graphs against Textual Sources](https://mlsanity.com/paper/2210.14846/) (2022)
- [Creating an African American-Sounding TTS: Guidelines, Technical Challenges,and Surprising Evaluations](https://mlsanity.com/paper/2403.11209/) (2024)
- [Leveraging Human Feedback to Scale Educational Datasets: Combining Crowdworkers and Comparative Judgement](https://mlsanity.com/paper/2305.12894/) (2023)
- [Detecting The Corruption Of Online Questionnaires By Artificial Intelligence](https://mlsanity.com/paper/2308.07499/) (2023)
- [HumanAL: Calibrating Human Matching Beyond a Single Task](https://mlsanity.com/paper/2205.03209/) (2022)
- [Trustworthy Human Computation: A Survey](https://mlsanity.com/paper/2210.12324/) (2022)
- [Handwritten Text Recognition from Crowdsourced Annotations](https://mlsanity.com/paper/2306.10878/) (2023)
- [Ensuring Reliable Participation in Subjective Video Quality Tests Across Platforms](https://mlsanity.com/paper/2509.20001/) (2025)
- [Overcoming the Incentive Collapse Paradox](https://mlsanity.com/paper/2603.27049/) (2026)

## Abstract (arXiv)

Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar - all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.