# Variability Matters : Evaluating inter-rater variability in histopathology for robust cell detection

arXiv: https://arxiv.org/abs/2210.05175 · HTML: https://mlsanity.com/paper/2210.05175/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 38 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Learning to Learn from Noisy Labeled Data](https://mlsanity.com/paper/1812.05214/) (2018)
- [Learning From Noisy Large-Scale Datasets With Minimal Supervision](https://mlsanity.com/paper/1701.01619/) (2017)
- [A Generalized Deep Learning Framework for Whole-Slide Image Segmentation and Analysis](https://mlsanity.com/paper/2001.00258/) (2020)
- [NuCLS: A scalable crowdsourcing, deep learning approach and dataset for nucleus classification, localization and segmentation](https://mlsanity.com/paper/2102.09099/) (2021)
- [Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants](https://mlsanity.com/paper/2112.09062/) (2021)
- [Iterative annotation to ease neural network training: Specialized machine learning in medical image analysis](https://mlsanity.com/paper/1812.07509/) (2018)
- [Let's agree to disagree: learning highly debatable multirater labelling](https://mlsanity.com/paper/1909.01891/) (2019)
- [Early Experiences with Crowdsourcing Airway Annotations in Chest CT](https://mlsanity.com/paper/1706.02055/) (2017)
- [HEp-2 Cell Image Classification with Deep Convolutional Neural Networks](https://mlsanity.com/paper/1504.02531/) (2015)

## Built on by (most cited first, 3 of 3)

- [Temporal Inversion for Learning Interval Change in Chest X-Rays](https://mlsanity.com/paper/2604.04563/) (2026)
- [Atlas H&E-TME: Scalable AI-Based Tissue Profiling at Expert Pathologist-Level Accuracy](https://mlsanity.com/paper/2606.12346/) (2026)
- [PreMix: Label-Efficient Multiple Instance Learning via Non-Contrastive Pre-training and Feature Mixing](https://mlsanity.com/paper/2408.01162/) (2024)

## Abstract (arXiv)

Large annotated datasets have been a key component in the success of deep learning. However, annotating medical images is challenging as it requires expertise and a large budget. In particular, annotating different types of cells in histopathology suffer from high inter- and intra-rater variability due to the ambiguity of the task. Under this setting, the relation between annotators' variability and model performance has received little attention. We present a large-scale study on the variability of cell annotations among 120 board-certified pathologists and how it affects the performance of a deep learning model. We propose a method to measure such variability, and by excluding those annotators with low variability, we verify the trade-off between the amount of data and its quality. We found that naively increasing the data size at the expense of inter-rater variability does not necessarily lead to better-performing models in cell detection. Instead, decreasing the inter-rater variability with the expense of decreasing dataset size increased the model performance. Furthermore, models trained from data annotated with lower inter-labeler variability outperform those from higher inter-labeler variability. These findings suggest that the evaluation of the annotators may help tackle the fundamental budget issues in the histopathology domain