# Crowdsourcing in Computer Vision

arXiv: https://arxiv.org/abs/1611.02145 · HTML: https://mlsanity.com/paper/1611.02145/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2016.

## Built on by (most cited first, 30 of 31)

- [A comprehensive survey on deep active learning in medical image analysis](https://mlsanity.com/paper/2310.14230/) (2023)
- [NuCLS: A scalable crowdsourcing, deep learning approach and dataset for nucleus classification, localization and segmentation](https://mlsanity.com/paper/2102.09099/) (2021)
- [A Survey on Deep Learning for Skin Lesion Segmentation](https://mlsanity.com/paper/2206.00356/) (2022)
- [From colouring-in to pointillism: revisiting semantic segmentation supervision](https://mlsanity.com/paper/2210.14142/) (2022)
- [Conformalized Credal Set Predictors](https://mlsanity.com/paper/2402.10723/) (2024)
- [Streaming Active Learning with Deep Neural Networks](https://mlsanity.com/paper/2303.02535/) (2023)
- [Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets](https://mlsanity.com/paper/2104.12690/) (2021)
- [Decoupling the Class Label and the Target Concept in Machine Unlearning](https://mlsanity.com/paper/2406.08288/) (2024)
- [Knowledge Learning with Crowdsourcing: A Brief Review and Systematic Perspective](https://mlsanity.com/paper/2206.09315/) (2022)
- [Towards Better User Studies in Computer Graphics and Vision](https://mlsanity.com/paper/2206.11461/) (2022)
- [Active Learning for Improved Semi-Supervised Semantic Segmentation in Satellite Images](https://mlsanity.com/paper/2110.07782/) (2021)
- [ALBench: A Framework for Evaluating Active Learning in Object Detection](https://mlsanity.com/paper/2207.13339/) (2022)
- [Evaluating AI systems under uncertain ground truth: a case study in dermatology](https://mlsanity.com/paper/2307.02191/) (2023)
- [Augmenting Chest X-ray Datasets with Non-Expert Annotations](https://mlsanity.com/paper/2309.02244/) (2023)
- [MEAL: Manifold Embedding-based Active Learning](https://mlsanity.com/paper/2106.11858/) (2021)
- [In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workers](https://mlsanity.com/paper/2112.02255/) (2021)
- [Active Measurement: Efficient Estimation at Scale](https://mlsanity.com/paper/2507.01372/) (2025)
- [Improved Adaptive Algorithm for Scalable Active Learning with Weak Labeler](https://mlsanity.com/paper/2211.02233/) (2022)
- [Combating Exacerbated Heterogeneity for Robust Models in Federated Learning](https://mlsanity.com/paper/2303.00250/) (2023)
- [EventAnchor: Reducing Human Interactions in Event Annotation of Racket Sports Videos](https://mlsanity.com/paper/2101.04954/) (2021)
- [Optimal Conformal Prediction under Epistemic Uncertainty](https://mlsanity.com/paper/2505.19033/) (2025)
- [Multiview Representation Learning from Crowdsourced Triplet Comparisons](https://mlsanity.com/paper/2302.03987/) (2023)
- [A Generalizable Light Transport 3D Embedding for Global Illumination](https://mlsanity.com/paper/2510.18189/) (2025)
- [Incentive-Tuning: Understanding and Designing Incentives for Empirical Human-AI Decision-Making Studies](https://mlsanity.com/paper/2601.15064/) (2026)
- [Semiotically-grounded distant viewing of diagrams: insights from two multimodal corpora](https://mlsanity.com/paper/2103.04692/) (2021)
- [Resolving the Human Subjects Status of Machine Learning's Crowdworkers](https://mlsanity.com/paper/2206.04039/) (2022)
- [Manifold Path Guiding for Importance Sampling Specular Chains](https://mlsanity.com/paper/2311.12818/) (2023)
- [Fine-Grained Counting with Crowd-Sourced Supervision](https://mlsanity.com/paper/2205.11398/) (2022)
- [Design and Evaluation of Camera-Centric Mobile Crowdsourcing Applications](https://mlsanity.com/paper/2409.03012/) (2024)
- [From Label Error Detection to Correction: A Modular Framework and Benchmark for Object Detection Datasets](https://mlsanity.com/paper/2508.06556/) (2025)

## Abstract (arXiv)

Computer vision systems require large amounts of manually annotated data to properly learn challenging visual concepts. Crowdsourcing platforms offer an inexpensive method to capture human knowledge and understanding, for a vast number of visual perception tasks. In this survey, we describe the types of annotations computer vision researchers have collected using crowdsourcing, and how they have ensured that this data is of high quality while annotation effort is minimized. We begin by discussing data collection on both classic (e.g., object recognition) and recent (e.g., visual story-telling) vision tasks. We then summarize key design decisions for creating effective data collection interfaces and workflows, and present strategies for intelligently selecting the most important data instances to annotate. Finally, we conclude with some thoughts on the future of crowdsourcing in computer vision.