# Learning from Multiple Expert Annotators for Enhancing Anomaly Detection in Medical Image Analysis

arXiv: https://arxiv.org/abs/2203.10611 · HTML: https://mlsanity.com/paper/2203.10611/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 62 references that are in the map.
- First posted on arXiv 20 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [Object Detection in 20 Years: A Survey](https://mlsanity.com/paper/1905.05055/) (2019)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations](https://mlsanity.com/paper/2012.15029/) (2020)
- [Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion](https://mlsanity.com/paper/1902.03680/) (2019)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [Deep Learning for Medical Image Processing: Overview, Challenges and Future](https://mlsanity.com/paper/1704.06825/) (2017)
- [COVID-CT-Dataset: A CT Scan Dataset about COVID-19](https://mlsanity.com/paper/2003.13865/) (2020)
- [Disentangling Human Error from the Ground Truth in Segmentation of Medical Images](https://mlsanity.com/paper/2007.15963/) (2020)
- [VinDr-RibCXR: A Benchmark Dataset for Automatic Segmentation and Labeling of Individual Ribs on Chest X-rays](https://mlsanity.com/paper/2107.01327/) (2021)
- [VinDr-SpineXR: A deep learning framework for spinal lesions detection and classification from radiographs](https://mlsanity.com/paper/2106.12930/) (2021)
- [DICOM Imaging Router: An Open Deep Learning Framework for Classification of Body Parts from DICOM X-ray Scans](https://mlsanity.com/paper/2108.06490/) (2021)
- [A Technical Survey on Statistical Modelling and Design Methods for Crowdsourcing Quality Control](https://mlsanity.com/paper/1812.02736/) (2018)
- [Learning from Multiple Annotators by Incorporating Instance Features](https://mlsanity.com/paper/2106.15146/) (2021)

## Built on by (most cited first, 13 of 13)

- [Active Policy Improvement from Multiple Black-box Oracles](https://mlsanity.com/paper/2306.10259/) (2023)
- [An Accurate and Explainable Deep Learning System Improves Interobserver Agreement in the Interpretation of Chest Radiograph](https://mlsanity.com/paper/2208.03545/) (2022)
- [Label Convergence: Defining an Upper Performance Bound in Object Recognition through Contradictory Annotations](https://mlsanity.com/paper/2409.09412/) (2024)
- [Learning to diagnose common thorax diseases on chest radiographs from radiology reports in Vietnamese](https://mlsanity.com/paper/2209.04794/) (2022)
- [Drawing the Same Bounding Box Twice? Coping Noisy Annotations in Object Detection with Repeated Labels](https://mlsanity.com/paper/2309.09742/) (2023)
- [Bayesian Detector Combination for Object Detection with Crowdsourced Annotations](https://mlsanity.com/paper/2407.07958/) (2024)
- [Exemplar Med-DETR: Toward Generalized and Robust Lesion Detection in Mammogram Images and beyond](https://mlsanity.com/paper/2507.19621/) (2025)
- [Improving Object Detection in Medical Image Analysis through Multiple Expert Annotators: An Empirical Investigation](https://mlsanity.com/paper/2303.16507/) (2023)
- [Evaluating the impact of an explainable machine learning system on the interobserver agreement in chest radiograph interpretation](https://mlsanity.com/paper/2304.01220/) (2023)
- [Context Enhancement with Reconstruction as Sequence for Unified Unsupervised Anomaly Detection](https://mlsanity.com/paper/2409.06285/) (2024)
- [Multi-Cue Anomaly Detection and Localization under Data Contamination](https://mlsanity.com/paper/2601.22913/) (2026)
- [Calibrating Probabilistic Object Detectors with Annotator Disagreement](https://mlsanity.com/paper/2605.24722/) (2026)
- [AUC Maximization from Biased Positive-unlabeled Data with Confidence](https://mlsanity.com/paper/2609.10928/) (2026)

## Abstract (arXiv)

Building an accurate computer-aided diagnosis system based on data-driven approaches requires a large amount of high-quality labeled data. In medical imaging analysis, multiple expert annotators often produce subjective estimates about "ground truth labels" during the annotation process, depending on their expertise and experience. As a result, the labeled data may contain a variety of human biases with a high rate of disagreement among annotators, which significantly affect the performance of supervised machine learning algorithms. To tackle this challenge, we propose a simple yet effective approach to combine annotations from multiple radiology experts for training a deep learning-based detector that aims to detect abnormalities on medical scans. The proposed method first estimates the ground truth annotations and confidence scores of training examples. The estimated annotations and their scores are then used to train a deep learning detector with a re-weighted loss function to localize abnormal findings. We conduct an extensive experimental evaluation of the proposed approach on both simulated and real-world medical imaging datasets. The experimental results show that our approach significantly outperforms baseline approaches that do not consider the disagreements among annotators, including methods in which all of the noisy annotations are treated equally as ground truth and the ensemble of different models trained on different label sets provided separately by annotators.