# Embracing Error to Enable Rapid Crowdsourcing

arXiv: https://arxiv.org/abs/1602.04506 · HTML: https://mlsanity.com/paper/1602.04506/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2016.

## Built on by (most cited first, 15 of 15)

- [The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://mlsanity.com/paper/2211.02570/) (2022)
- [RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data](https://mlsanity.com/paper/2210.12634/) (2022)
- [Perception Tokens Enhance Visual Reasoning in Multimodal Language Models](https://mlsanity.com/paper/2412.03548/) (2024)
- [Investigating Why Contrastive Learning Benefits Robustness Against Label Noise](https://mlsanity.com/paper/2201.12498/) (2022)
- [Dissonance Between Human and Machine Understanding](https://mlsanity.com/paper/2101.07337/) (2021)
- [Assessing the Quality of the Datasets by Identifying Mislabeled Samples](https://mlsanity.com/paper/2109.05000/) (2021)
- [EventAnchor: Reducing Human Interactions in Event Annotation of Racket Sports Videos](https://mlsanity.com/paper/2101.04954/) (2021)
- [Sketch-based Creativity Support Tools using Deep Learning](https://mlsanity.com/paper/2111.09991/) (2021)
- [Investigating the Impact of Model Width and Density on Generalization in Presence of Label Noise](https://mlsanity.com/paper/2208.08003/) (2022)
- [LNL+K: Enhancing Learning with Noisy Labels Through Noise Source Knowledge Integration](https://mlsanity.com/paper/2306.11911/) (2023)
- [AlleNoise: large-scale text classification benchmark dataset with real-world label noise](https://mlsanity.com/paper/2407.10992/) (2024)
- [Visual Intelligence through Human Interaction](https://mlsanity.com/paper/2111.06913/) (2021)
- [TruEyes: Utilizing Microtasks in Mobile Apps for Crowdsourced Labeling of Machine Learning Datasets](https://mlsanity.com/paper/2209.14708/) (2022)
- [A Unified Framework for Connecting Noise Modeling to Boost Noise Detection](https://mlsanity.com/paper/2312.00827/) (2023)
- [Trading Human Curation for Synthetic Augmentation in RLVR](https://mlsanity.com/paper/2606.03800/) (2026)

## Abstract (arXiv)

Microtask crowdsourcing has enabled dataset advances in social science and machine learning, but existing crowdsourcing schemes are too expensive to scale up with the expanding volume of data. To scale and widen the applicability of crowdsourcing, we present a technique that produces extremely rapid judgments for binary and categorical labels. Rather than punishing all errors, which causes workers to proceed slowly and deliberately, our technique speeds up workers' judgments to the point where errors are acceptable and even expected. We demonstrate that it is possible to rectify these errors by randomizing task order and modeling response latency. We evaluate our technique on a breadth of common labeling tasks such as image verification, word similarity, sentiment analysis and topic classification. Where prior work typically achieves a 0.25x to 1x speedup over fixed majority vote, our approach often achieves an order of magnitude (10x) speedup.