# Labeling instructions matter in biomedical image analysis

arXiv: https://arxiv.org/abs/2207.09899 · HTML: https://mlsanity.com/paper/2207.09899/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 68 references that are in the map.
- First posted on arXiv 20 Jul 2022.

## Builds on (most cited first, 21 of 21)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [Deep learning with noisy labels: exploring techniques and remedies in medical image analysis](https://mlsanity.com/paper/1912.02911/) (2019)
- [Surgical Data Science -- from Concepts toward Clinical Translation](https://mlsanity.com/paper/2011.02284/) (2020)
- [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018)
- [Common Limitations of Image Processing Metrics: A Picture Story](https://mlsanity.com/paper/2104.05642/) (2021)
- [BIAS: Transparent reporting of biomedical image analysis challenges](https://mlsanity.com/paper/1910.04071/) (2019)
- [Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotation](https://mlsanity.com/paper/2112.04554/) (2021)
- [Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Papers](https://mlsanity.com/paper/2108.02922/) (2021)
- [Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Room](https://mlsanity.com/paper/2005.03501/) (2020)
- [A Survey of Crowdsourcing in Medical Image Analysis](https://mlsanity.com/paper/1902.09159/) (2019)
- [An Empirical Study into Annotator Agreement, Ground Truth Estimation, and Algorithm Evaluation](https://mlsanity.com/paper/1307.0426/) (2013)
- [The challenges of deploying artificial intelligence models in a rapidly evolving pandemic](https://mlsanity.com/paper/2005.12137/) (2020)
- [Medical Imaging and Machine Learning](https://mlsanity.com/paper/2103.01938/) (2021)
- [Easy, Reproducible and Quality-Controlled Data Collection with Crowdaq](https://mlsanity.com/paper/2010.06694/) (2020)
- [PSA-Net: Deep Learning based Physician Style-Aware Segmentation Network for Post-Operative Prostate Cancer Clinical Target Volume](https://mlsanity.com/paper/2102.07880/) (2021)
- [Early Experiences with Crowdsourcing Airway Annotations in Chest CT](https://mlsanity.com/paper/1706.02055/) (2017)
- [How can we learn (more) from challenges? A statistical approach to driving future algorithm development](https://mlsanity.com/paper/2106.09302/) (2021)
- [Can non-specialists provide high quality gold standard labels in challenging modalities?](https://mlsanity.com/paper/2107.14682/) (2021)

## Built on by (most cited first, 20 of 20)

- [The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CT](https://mlsanity.com/paper/2307.01984/) (2023)
- [Unleashing the Strengths of Unlabeled Data in Pan-cancer Abdominal Organ Quantification: the FLARE22 Challenge](https://mlsanity.com/paper/2308.05862/) (2023)
- [A comprehensive survey on deep active learning in medical image analysis](https://mlsanity.com/paper/2310.14230/) (2023)
- [Approaching Peak Ground Truth](https://mlsanity.com/paper/2301.00243/) (2022)
- [Real World Federated Learning with a Knowledge Distilled Transformer for Cardiac CT Imaging](https://mlsanity.com/paper/2407.07557/) (2024)
- [In the Picture: Medical Imaging Datasets, Artifacts, and their Living Review](https://mlsanity.com/paper/2501.10727/) (2025)
- [Quality Assured: Rethinking Annotation Strategies in Imaging AI](https://mlsanity.com/paper/2407.17596/) (2024)
- [FUNAvg: Federated Uncertainty Weighted Averaging for Datasets with Diverse Labels](https://mlsanity.com/paper/2407.07488/) (2024)
- [FetMRQC: a robust quality control system for multi-centric fetal brain MRI](https://mlsanity.com/paper/2311.04780/) (2023)
- [LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems](https://mlsanity.com/paper/2403.09810/) (2024)
- [Attribute Annotation and Bias Evaluation in Visual Datasets for Autonomous Driving](https://mlsanity.com/paper/2312.06306/) (2023)
- [Large-scale cervical precancerous screening via AI-assisted cytology whole slide image analysis](https://mlsanity.com/paper/2407.19512/) (2024)
- [Improving Task Instructions for Data Annotators: How Clear Rules and Higher Pay Increase Performance in Data Annotation in the AI Economy](https://mlsanity.com/paper/2312.14565/) (2023)
- [A Bayesian Approach to Segmentation with Noisy Labels via Spatially Correlated Distributions](https://mlsanity.com/paper/2504.14795/) (2025)
- [Decomposition Sampling for Efficient Region Annotations in Active Learning](https://mlsanity.com/paper/2512.07606/) (2025)
- [orGAN: A Synthetic Data Augmentation Pipeline for Simultaneous Generation of Surgical Images and Ground Truth Labels](https://mlsanity.com/paper/2506.14303/) (2025)
- [Active Transfer Bagging: A New Approach for Accelerated Active Learning Acquisition of Data by Combined Transfer Learning and Bagging Based Models](https://mlsanity.com/paper/2602.02415/) (2026)
- [XAI-CLIP: ROI-Guided Perturbation Framework for Explainable Medical Image Segmentation in Multimodal Vision-Language Models](https://mlsanity.com/paper/2602.07017/) (2026)
- [Enhancing MedSAM with a Lightweight Box Predictor for Medical Image Segmentation](https://mlsanity.com/paper/2606.04705/) (2026)
- [A Multi-Annotator Study of Segmentation Noise and Uncertainty in Turbid Underwater Images](https://mlsanity.com/paper/2608.15363/) (2026)

## Abstract (arXiv)

Biomedical image analysis algorithm validation depends on high-quality annotation of reference datasets, for which labeling instructions are key. Despite their importance, their optimization remains largely unexplored. Here, we present the first systematic study of labeling instructions and their impact on annotation quality in the field. Through comprehensive examination of professional practice and international competitions registered at the MICCAI Society, we uncovered a discrepancy between annotators' needs for labeling instructions and their current quality and availability. Based on an analysis of 14,040 images annotated by 156 annotators from four professional companies and 708 Amazon Mechanical Turk (MTurk) crowdworkers using instructions with different information density levels, we further found that including exemplary images significantly boosts annotation performance compared to text-only descriptions, while solely extending text descriptions does not. Finally, professional annotators constantly outperform MTurk crowdworkers. Our study raises awareness for the need of quality standards in biomedical image analysis labeling instructions.