# Quality Assured: Rethinking Annotation Strategies in Imaging AI

arXiv: https://arxiv.org/abs/2407.17596 · HTML: https://mlsanity.com/paper/2407.17596/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 49 references that are in the map.
- First posted on arXiv 24 Jul 2024.

## Builds on (most cited first, 18 of 18)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming](https://mlsanity.com/paper/1907.07484/) (2019)
- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [Platypus: Quick, Cheap, and Powerful Refinement of LLMs](https://mlsanity.com/paper/2308.07317/) (2023)
- [Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks](https://mlsanity.com/paper/2306.07899/) (2023)
- [Understanding metric-related pitfalls in image analysis validation](https://mlsanity.com/paper/2302.01790/) (2023)
- [Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Room](https://mlsanity.com/paper/2005.03501/) (2020)
- [ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations](https://mlsanity.com/paper/2211.01866/) (2022)
- [A Survey of Crowdsourcing in Medical Image Analysis](https://mlsanity.com/paper/1902.09159/) (2019)
- [Labeling instructions matter in biomedical image analysis](https://mlsanity.com/paper/2207.09899/) (2022)
- [Difficulty-aware Glaucoma Classification with Multi-Rater Consensus Modeling](https://mlsanity.com/paper/2007.14848/) (2020)
- [The Data-Production Dispositif](https://mlsanity.com/paper/2205.11963/) (2022)
- [Early Experiences with Crowdsourcing Airway Annotations in Chest CT](https://mlsanity.com/paper/1706.02055/) (2017)
- [The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help](https://mlsanity.com/paper/2105.09457/) (2021)
- [How can we learn (more) from challenges? A statistical approach to driving future algorithm development](https://mlsanity.com/paper/2106.09302/) (2021)
- [Neglected Free Lunch -- Learning Image Classifiers Using Annotation Byproducts](https://mlsanity.com/paper/2303.17595/) (2023)
- [Prototype Tasks: Improving Crowdsourcing Results through Rapid, Iterative Task Design](https://mlsanity.com/paper/1707.05645/) (2017)
- [Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Dataset Labeling](https://mlsanity.com/paper/2306.06773/) (2023)

## Built on by (most cited first, 6 of 6)

- [OpenLex3D: A Tiered Evaluation Benchmark for Open-Vocabulary 3D Scene Representations](https://mlsanity.com/paper/2503.19764/) (2025)
- [SegAssess: Panoramic quality mapping for robust and transferable unsupervised segmentation assessment](https://mlsanity.com/paper/2509.01183/) (2025)
- [Position: Early-Stage Quality Assurance in Annotation Pipelines Is More Cost-Effective Than Late-Stage Validation](https://mlsanity.com/paper/2605.15714/) (2026)
- [Cost Savings from Automatic Quality Assessment of Generated Images](https://mlsanity.com/paper/2510.16179/) (2025)
- [Region-Normalized DPO for Medical Image Segmentation under Noisy Judges](https://mlsanity.com/paper/2601.23222/) (2026)
- [Decoupled Single-Mask Annotation Noise Detection via Cross-Sectional Patch Self-Consistency](https://mlsanity.com/paper/2607.05965/) (2026)

## Abstract (arXiv)

This paper does not describe a novel method. Instead, it studies an essential foundation for reliable benchmarking and ultimately real-world application of AI-based image analysis: generating high-quality reference annotations. Previous research has focused on crowdsourcing as a means of outsourcing annotations. However, little attention has so far been given to annotation companies, specifically regarding their internal quality assurance (QA) processes. Therefore, our aim is to evaluate the influence of QA employed by annotation companies on annotation quality and devise methodologies for maximizing data annotation efficacy. Based on a total of 57,648 instance segmented images obtained from a total of 924 annotators and 34 QA workers from four annotation companies and Amazon Mechanical Turk (MTurk), we derived the following insights: (1) Annotation companies perform better both in terms of quantity and quality compared to the widely used platform MTurk. (2) Annotation companies' internal QA only provides marginal improvements, if any. However, improving labeling instructions instead of investing in QA can substantially boost annotation performance. (3) The benefit of internal QA depends on specific image characteristics. Our work could enable researchers to derive substantially more value from a fixed annotation budget and change the way annotation companies conduct internal QA.