# A Survey of Crowdsourcing in Medical Image Analysis

arXiv: https://arxiv.org/abs/1902.09159 · HTML: https://mlsanity.com/paper/1902.09159/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Feb 2019.

## Built on by (most cited first, 20 of 20)

- [How I failed machine learning in medical imaging -- shortcomings and recommendations](https://mlsanity.com/paper/2103.10292/) (2021)
- [Towards Transparency in Dermatology Image Datasets with Skin Tone Annotations by Experts, Crowds, and an Algorithm](https://mlsanity.com/paper/2207.02942/) (2022)
- [NuCLS: A scalable crowdsourcing, deep learning approach and dataset for nucleus classification, localization and segmentation](https://mlsanity.com/paper/2102.09099/) (2021)
- [Q-Net: Query-Informed Few-Shot Medical Image Segmentation](https://mlsanity.com/paper/2208.11451/) (2022)
- [Labeling instructions matter in biomedical image analysis](https://mlsanity.com/paper/2207.09899/) (2022)
- [Vessel-CAPTCHA: an efficient learning framework for vessel annotation and segmentation](https://mlsanity.com/paper/2101.09321/) (2021)
- [Label noise in segmentation networks : mitigation must deal with bias](https://mlsanity.com/paper/2107.02189/) (2021)
- [Diversified and Personalized Multi-rater Medical Image Segmentation](https://mlsanity.com/paper/2403.13417/) (2024)
- [Weak Supervision for Affordable Modeling of Electrocardiogram Data](https://mlsanity.com/paper/2201.02936/) (2022)
- [In the Picture: Medical Imaging Datasets, Artifacts, and their Living Review](https://mlsanity.com/paper/2501.10727/) (2025)
- [Augmenting Chest X-ray Datasets with Non-Expert Annotations](https://mlsanity.com/paper/2309.02244/) (2023)
- [Quality Assured: Rethinking Annotation Strategies in Imaging AI](https://mlsanity.com/paper/2407.17596/) (2024)
- [Terabyte-scale supervised 3D training and benchmarking dataset of the mouse kidney](https://mlsanity.com/paper/2108.02226/) (2021)
- [Active Label Refinement for Robust Training of Imbalanced Medical Image Classification Tasks in the Presence of High Label Noise](https://mlsanity.com/paper/2407.05973/) (2024)
- [Improving Task Instructions for Data Annotators: How Clear Rules and Higher Pay Increase Performance in Data Annotation in the AI Economy](https://mlsanity.com/paper/2312.14565/) (2023)
- [How does self-supervised pretraining improve robustness against noisy labels across various medical image classification datasets?](https://mlsanity.com/paper/2401.07990/) (2024)
- [Advancing Histopathology with Deep Learning Under Data Scarcity: A Decade in Review](https://mlsanity.com/paper/2410.19820/) (2024)
- [Rethinking Generalization Performance of Surgical Phase Recognition with Expert-Generated Annotations](https://mlsanity.com/paper/2110.11626/) (2021)
- [Crowdsourced human-based computational approach for tagging peripheral blood smear sample images from Sickle Cell Disease patients using non-expert users](https://mlsanity.com/paper/2501.07196/) (2025)
- [Active Label Cleaning for Reliable Detection of Electron Dense Deposits in Transmission Electron Microscopy Images](https://mlsanity.com/paper/2602.05250/) (2026)

## Abstract (arXiv)

Rapid advances in image processing capabilities have been seen across many domains, fostered by the application of machine learning algorithms to "big-data". However, within the realm of medical image analysis, advances have been curtailed, in part, due to the limited availability of large-scale, well-annotated datasets. One of the main reasons for this is the high cost often associated with producing large amounts of high-quality meta-data. Recently, there has been growing interest in the application of crowdsourcing for this purpose; a technique that has proven effective for creating large-scale datasets across a range of disciplines, from computer vision to astrophysics. Despite the growing popularity of this approach, there has not yet been a comprehensive literature review to provide guidance to researchers considering using crowdsourcing methodologies in their own medical imaging analysis. In this survey, we review studies applying crowdsourcing to the analysis of medical images, published prior to July 2018. We identify common approaches, challenges and considerations, providing guidance of utility to researchers adopting this approach. Finally, we discuss future opportunities for development within this emerging domain.