# Did You Get What You Paid For? Rethinking Annotation Cost of Deep Learning Based Computer Aided Detection in Chest Radiographs

arXiv: https://arxiv.org/abs/2209.15314 · HTML: https://mlsanity.com/paper/2209.15314/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 15 references that are in the map.
- First posted on arXiv 30 Sep 2022.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations](https://mlsanity.com/paper/1707.03237/) (2017)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Deep learning with noisy labels: exploring techniques and remedies in medical image analysis](https://mlsanity.com/paper/1912.02911/) (2019)
- [Deep Learning for Chest X-ray Analysis: A Survey](https://mlsanity.com/paper/2103.08700/) (2021)
- [Self-Loop Uncertainty: A Novel Pseudo-Label for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2007.09854/) (2020)
- [Exploring large scale public medical image datasets](https://mlsanity.com/paper/1907.12720/) (2019)
- [Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models](https://mlsanity.com/paper/2103.00429/) (2021)
- [Caveats in Generating Medical Imaging Labels from Radiology Reports](https://mlsanity.com/paper/1905.02283/) (2019)

## Built on by (most cited first, 5 of 5)

- [Exploring Image Augmentations for Siamese Representation Learning with Chest X-Rays](https://mlsanity.com/paper/2301.12636/) (2023)
- [Augmenting Chest X-ray Datasets with Non-Expert Annotations](https://mlsanity.com/paper/2309.02244/) (2023)
- [Augmentation-aware Self-supervised Learning with Conditioned Projector](https://mlsanity.com/paper/2306.06082/) (2023)
- [Optimizing the Procedure of CT Segmentation Labeling](https://mlsanity.com/paper/2303.14089/) (2023)
- [Unsupervised Domain Adaptation Using Compact Internal Representations](https://mlsanity.com/paper/2401.07207/) (2024)

## Abstract (arXiv)

As deep networks require large amounts of accurately labeled training data, a strategy to collect sufficiently large and accurate annotations is as important as innovations in recognition methods. This is especially true for building Computer Aided Detection (CAD) systems for chest X-rays where domain expertise of radiologists is required to annotate the presence and location of abnormalities on X-ray images. However, there lacks concrete evidence that provides guidance on how much resource to allocate for data annotation such that the resulting CAD system reaches desired performance. Without this knowledge, practitioners often fall back to the strategy of collecting as much detail as possible on as much data as possible which is cost inefficient. In this work, we investigate how the cost of data annotation ultimately impacts the CAD model performance on classification and segmentation of chest abnormalities in frontal-view X-ray images. We define the cost of annotation with respect to the following three dimensions: quantity, quality and granularity of labels. Throughout this study, we isolate the impact of each dimension on the resulting CAD model performance on detecting 10 chest abnormalities in X-rays. On a large scale training data with over 120K X-ray images with gold-standard annotations, we find that cost-efficient annotations provide great value when collected in large amounts and lead to competitive performance when compared to models trained with only gold-standard annotations. We also find that combining large amounts of cost efficient annotations with only small amounts of expensive labels leads to competitive CAD models at a much lower cost.