# Early Experiences with Crowdsourcing Airway Annotations in Chest CT

arXiv: https://arxiv.org/abs/1706.02055 · HTML: https://mlsanity.com/paper/1706.02055/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2017.

## Built on by (most cited first, 6 of 6)

- [Labeling instructions matter in biomedical image analysis](https://mlsanity.com/paper/2207.09899/) (2022)
- [Vessel-CAPTCHA: an efficient learning framework for vessel annotation and segmentation](https://mlsanity.com/paper/2101.09321/) (2021)
- [Quality Assured: Rethinking Annotation Strategies in Imaging AI](https://mlsanity.com/paper/2407.17596/) (2024)
- [Can non-specialists provide high quality gold standard labels in challenging modalities?](https://mlsanity.com/paper/2107.14682/) (2021)
- [Variability Matters : Evaluating inter-rater variability in histopathology for robust cell detection](https://mlsanity.com/paper/2210.05175/) (2022)
- [An overview of deep learning in medical imaging](https://mlsanity.com/paper/2202.08546/) (2022)

## Abstract (arXiv)

Measuring airways in chest computed tomography (CT) images is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated data to perform well. We investigate whether crowdsourcing can be used to gather airway annotations which can serve directly for measuring the airways, or as training data for the algorithms. We generate image slices at known locations of airways and request untrained crowd workers to outline the airway lumen and airway wall. Our results show that the workers are able to interpret the images, but that the instructions are too complex, leading to many unusable annotations. After excluding unusable annotations, quantitative results show medium to high correlations with expert measurements of the airways. Based on this positive experience, we describe a number of further research directions and provide insight into the challenges of crowdsourcing in medical images from the perspective of first-time users.