mlsanityData to 7 Oct 2026

Labeling instructions matter in biomedical image analysis

arXiv 2207.09899 ↗First posted 20 Jul 2022cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

Biomedical image analysis algorithm validation depends on high-quality annotation of reference datasets, for which labeling instructions are key. Despite their importance, their optimization remains largely unexplored. Here, we present the first systematic study of labeling instructions and their impact on annotation quality in the field. Through comprehensive examination of professional practice and international competitions registered at the MICCAI Society, we uncovered a discrepancy between annotators' needs for labeling instructions and their current quality and availability. Based on an analysis of 14,040 images annotated by 156 annotators from four professional companies and 708 Amazon Mechanical Turk (MTurk) crowdworkers using instructions with different information density levels, we further found that including exemplary images significantly boosts annotation performance compared to text-only descriptions, while solely extending text descriptions does not. Finally, professional annotators constantly outperform MTurk crowdworkers. Our study raises awareness for the need of quality standards in biomedical image analysis labeling instructions.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 21 of its 68 references that are in the map. First posted on arXiv 20 Jul 2022.

Builds on

21 of 68 references.

References · 21

  1. 2018Datasheets for Datasetscited by 1,371
  2. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  3. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  4. 2022Metrics reloaded: Recommendations for image analysis validationcited by 190
  5. 2019Deep learning with noisy labels: exploring techniques and remedies in medical image analysiscited by 173
  6. 2020Surgical Data Science -- from Concepts toward Clinical Translationcited by 105
  7. 2018Why rankings of biomedical image analysis competitions should be interpreted with carecited by 90
  8. 2021Common Limitations of Image Processing Metrics: A Picture Storycited by 58
  9. 2019BIAS: Transparent reporting of biomedical image analysis challengescited by 57
  10. 2021Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotationcited by 51
  11. 2021Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Paperscited by 49
  12. 2020Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Roomcited by 47
9 more
  1. 2019A Survey of Crowdsourcing in Medical Image Analysiscited by 20
  2. 2013An Empirical Study into Annotator Agreement, Ground Truth Estimation, and Algorithm Evaluationcited by 15
  3. 2020The challenges of deploying artificial intelligence models in a rapidly evolving pandemiccited by 12
  4. 2021Medical Imaging and Machine Learningcited by 12
  5. 2020Easy, Reproducible and Quality-Controlled Data Collection with Crowdaqcited by 9
  6. 2021PSA-Net: Deep Learning based Physician Style-Aware Segmentation Network for Post-Operative Prostate Cancer Clinical Target Volumecited by 9
  7. 2017Early Experiences with Crowdsourcing Airway Annotations in Chest CTcited by 6
  8. 2021How can we learn (more) from challenges? A statistical approach to driving future algorithm developmentcited by 5
  9. 2021Can non-specialists provide high quality gold standard labels in challenging modalities?cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 5; 2025: 4; 2026: 4.

20 papers.

Sort:

By area:

  1. 2023The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CTcited by 101
  2. 2023Unleashing the Strengths of Unlabeled Data in Pan-cancer Abdominal Organ Quantification: the FLARE22 Challengecited by 50
  3. 2023A comprehensive survey on deep active learning in medical image analysiscited by 32
  4. 2022Approaching Peak Ground Truthcited by 8
  5. 2024Real World Federated Learning with a Knowledge Distilled Transformer for Cardiac CT Imagingcited by 8
  6. 2025In the Picture: Medical Imaging Datasets, Artifacts, and their Living Reviewcited by 8
  7. 2024Quality Assured: Rethinking Annotation Strategies in Imaging AIcited by 6
  8. 2024FUNAvg: Federated Uncertainty Weighted Averaging for Datasets with Diverse Labelscited by 5
  9. 2023FetMRQC: a robust quality control system for multi-centric fetal brain MRIcited by 4
  10. 2024LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systemscited by 4
  11. 2023Attribute Annotation and Bias Evaluation in Visual Datasets for Autonomous Drivingcited by 3
  12. 2024Large-scale cervical precancerous screening via AI-assisted cytology whole slide image analysiscited by 2
8 more
  1. 2023Improving Task Instructions for Data Annotators: How Clear Rules and Higher Pay Increase Performance in Data Annotation in the AI Economycited by 1
  2. 2025A Bayesian Approach to Segmentation with Noisy Labels via Spatially Correlated Distributionscited by 1
  3. 2025Decomposition Sampling for Efficient Region Annotations in Active Learningcited by 1
  4. 2025orGAN: A Synthetic Data Augmentation Pipeline for Simultaneous Generation of Surgical Images and Ground Truth Labelscited by 0
  5. 2026Active Transfer Bagging: A New Approach for Accelerated Active Learning Acquisition of Data by Combined Transfer Learning and Bagging Based Modelscited by 0
  6. 2026XAI-CLIP: ROI-Guided Perturbation Framework for Explainable Medical Image Segmentation in Multimodal Vision-Language Modelscited by 0
  7. 2026Enhancing MedSAM with a Lightweight Box Predictor for Medical Image Segmentationcited by 0
  8. 2026A Multi-Annotator Study of Segmentation Noise and Uncertainty in Turbid Underwater Imagescited by 0