# Let's agree to disagree: learning highly debatable multirater labelling

arXiv: https://arxiv.org/abs/1909.01891 · HTML: https://mlsanity.com/paper/1909.01891/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2019.

## Built on by (most cited first, 16 of 16)

- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [Where is VALDO? VAscular Lesions Detection and segmentatiOn challenge at MICCAI 2021](https://mlsanity.com/paper/2208.07167/) (2022)
- [Impact of individual rater style on deep learning uncertainty in medical imaging segmentation](https://mlsanity.com/paper/2105.02197/) (2021)
- [Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results](https://mlsanity.com/paper/2505.08685/) (2025)
- [Development and evaluation of intraoperative ultrasound segmentation with negative image frames and multiple observer labels](https://mlsanity.com/paper/2108.04114/) (2021)
- [Estimating MRI Image Quality via Image Reconstruction Uncertainty](https://mlsanity.com/paper/2106.10992/) (2021)
- [A Decoupled Uncertainty Model for MRI Segmentation Quality Estimation](https://mlsanity.com/paper/2109.02413/) (2021)
- [Opinions Vary? Diagnosis First!](https://mlsanity.com/paper/2202.06505/) (2022)
- [Calibrate the inter-observer segmentation uncertainty via diagnosis-first principle](https://mlsanity.com/paper/2208.03016/) (2022)
- [Variability Matters : Evaluating inter-rater variability in histopathology for robust cell detection](https://mlsanity.com/paper/2210.05175/) (2022)
- [Inter-Rater Uncertainty Quantification in Medical Image Segmentation via Rater-Specific Bayesian Neural Networks](https://mlsanity.com/paper/2306.16556/) (2023)
- [Dealing with Annotator Disagreement in Hate Speech Classification](https://mlsanity.com/paper/2502.08266/) (2025)
- [Bayesian Detector Combination for Object Detection with Crowdsourced Annotations](https://mlsanity.com/paper/2407.07958/) (2024)
- [Segmentation with Multiple Acceptable Annotations: A Case Study of Myocardial Segmentation in Contrast Echocardiography](https://mlsanity.com/paper/2106.15597/) (2021)
- [Good Enough? An Investigation on the Impact of Label Quality in Large-Scale Medical Datasets](https://mlsanity.com/paper/2505.20928/) (2025)
- [Assessing Pancreatic Ductal Adenocarcinoma Vascular Invasion: the PDACVI Benchmark](https://mlsanity.com/paper/2604.27582/) (2026)

## Abstract (arXiv)

Classification and differentiation of small pathological objects may greatly vary among human raters due to differences in training, expertise and their consistency over time. In a radiological setting, objects commonly have high within-class appearance variability whilst sharing certain characteristics across different classes, making their distinction even more difficult. As an example, markers of cerebral small vessel disease, such as enlarged perivascular spaces (EPVS) and lacunes, can be very varied in their appearance while exhibiting high inter-class similarity, making this task highly challenging for human raters. In this work, we investigate joint models of individual rater behaviour and multirater consensus in a deep learning setting, and apply it to a brain lesion object-detection task. Results show that jointly modelling both individual and consensus estimates leads to significant improvements in performance when compared to directly predicting consensus labels, while also allowing the characterization of human-rater consistency.