# Handling Inter-Annotator Agreement for Automated Skin Lesion Segmentation

arXiv: https://arxiv.org/abs/1906.02415 · HTML: https://mlsanity.com/paper/1906.02415/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2019.

## Built on by (most cited first, 9 of 9)

- [Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization](https://mlsanity.com/paper/2104.05833/) (2021)
- [Robust T-Loss for Medical Image Segmentation](https://mlsanity.com/paper/2306.00753/) (2023)
- [Modeling Annotator Preference and Stochastic Annotation Error for Medical Image Segmentation](https://mlsanity.com/paper/2111.13410/) (2021)
- [Label Convergence: Defining an Upper Performance Bound in Object Recognition through Contradictory Annotations](https://mlsanity.com/paper/2409.09412/) (2024)
- [CleanPatrick: A Benchmark for Image Data Cleaning](https://mlsanity.com/paper/2505.11034/) (2025)
- [What Can We Learn from Inter-Annotator Variability in Skin Lesion Segmentation?](https://mlsanity.com/paper/2508.09381/) (2025)
- [MSE-Nets: Multi-annotated Semi-supervised Ensemble Networks for Improving Segmentation of Medical Image with Ambiguous Boundaries](https://mlsanity.com/paper/2311.10380/) (2023)
- [KαLOS finds Consensus: A Meta-Algorithm for Evaluating Inter-Annotator Agreement in Complex Vision Tasks](https://mlsanity.com/paper/2603.27197/) (2026)
- [A Multi-Annotator Study of Segmentation Noise and Uncertainty in Turbid Underwater Images](https://mlsanity.com/paper/2608.15363/) (2026)

## Abstract (arXiv)

In this work, we explore the issue of the inter-annotator agreement for training and evaluating automated segmentation of skin lesions. We explore what different degrees of agreement represent, and how they affect different use cases for segmentation. We also evaluate how conditioning the ground truths using different (but very simple) algorithms may help to enhance agreement and may be appropriate for some use cases. The segmentation of skin lesions is a cornerstone task for automated skin lesion analysis, useful both as an end-result to locate/detect the lesions and as an ancillary task for lesion classification. Lesion segmentation, however, is a very challenging task, due not only to the challenge of image segmentation itself but also to the difficulty in obtaining properly annotated data. Detecting accurately the borders of lesions is challenging even for trained humans, since, for many lesions, those borders are fuzzy and ill-defined. Using lesions and annotations from the ISIC Archive, we estimate inter-annotator agreement for skin-lesion segmentation and propose several simple procedures that may help to improve inter-annotator agreement if used to condition the ground truths.