# An Empirical Study into Annotator Agreement, Ground Truth Estimation, and Algorithm Evaluation

arXiv: https://arxiv.org/abs/1307.0426 · HTML: https://mlsanity.com/paper/1307.0426/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2013.

## Built on by (most cited first, 16 of 16)

- [Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts](https://mlsanity.com/paper/2206.07948/) (2022)
- [Learning to Defer with Limited Expert Predictions](https://mlsanity.com/paper/2304.07306/) (2023)
- [A Survey on Deep Learning for Skin Lesion Segmentation](https://mlsanity.com/paper/2206.00356/) (2022)
- [Labeling instructions matter in biomedical image analysis](https://mlsanity.com/paper/2207.09899/) (2022)
- [Modeling Annotator Preference and Stochastic Annotation Error for Medical Image Segmentation](https://mlsanity.com/paper/2111.13410/) (2021)
- [Combating noisy labels in object detection datasets](https://mlsanity.com/paper/2211.13993/) (2022)
- [AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systems](https://mlsanity.com/paper/2402.06287/) (2024)
- [Learn Fine-grained Adaptive Loss for Multiple Anatomical Landmark Detection in Medical Images](https://mlsanity.com/paper/2105.09124/) (2021)
- [Learn2Agree: Fitting with Multiple Annotators without Objective Ground Truth](https://mlsanity.com/paper/2109.03596/) (2021)
- [Estimating Agreement by Chance for Sequence Annotation](https://mlsanity.com/paper/2407.11371/) (2024)
- [Interpreting Expert Annotation Differences in Animal Behavior](https://mlsanity.com/paper/2106.06114/) (2021)
- [Non-inferiority of Deep Learning Acute Ischemic Stroke Segmentation on Non-Contrast CT Compared to Expert Neuroradiologists](https://mlsanity.com/paper/2211.15341/) (2022)
- [A Bayesian Approach to Segmentation with Noisy Labels via Spatially Correlated Distributions](https://mlsanity.com/paper/2504.14795/) (2025)
- [Humanly Certifying Superhuman Classifiers](https://mlsanity.com/paper/2109.07867/) (2021)
- [A Multi-Annotator Study of Segmentation Noise and Uncertainty in Turbid Underwater Images](https://mlsanity.com/paper/2608.15363/) (2026)
- [Quantifying Annotation Agreement in Object Detection with Optimal Transport](https://mlsanity.com/paper/2610.19912/) (2026)

## Abstract (arXiv)

Although agreement between annotators has been studied in the past from a statistical viewpoint, little work has attempted to quantify the extent to which this phenomenon affects the evaluation of computer vision (CV) object detection algorithms. Many researchers utilise ground truth (GT) in experiments and more often than not this GT is derived from one annotator's opinion. How does the difference in opinion affect an algorithm's evaluation? Four examples of typical CV problems are chosen, and a methodology is applied to each to quantify the inter-annotator variance and to offer insight into the mechanisms behind agreement and the use of GT. It is found that when detecting linear objects annotator agreement is very low. The agreement in object position, linear or otherwise, can be partially explained through basic image properties. Automatic object detectors are compared to annotator agreement and it is found that a clear relationship exists. Several methods for calculating GTs from a number of annotations are applied and the resulting differences in the performance of the object detectors are quantified. It is found that the rank of a detector is highly dependent upon the method used to form the GT. It is also found that although the STAPLE and LSML GT estimation methods appear to represent the mean of the performance measured using the individual annotations, when there are few annotations, or there is a large variance in them, these estimates tend to degrade. Furthermore, one of the most commonly adopted annotation combination methods--consensus voting--accentuates more obvious features, which results in an overestimation of the algorithm's performance. Finally, it is concluded that in some datasets it may not be possible to state with any confidence that one algorithm outperforms another when evaluating upon one GT and a method for calculating confidence bounds is discussed.