# D-LEMA: Deep Learning Ensembles from Multiple Annotations -- Application to Skin Lesion Segmentation

arXiv: https://arxiv.org/abs/2012.07206 · HTML: https://mlsanity.com/paper/2012.07206/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2020.

## Built on by (most cited first, 16 of 16)

- [A Survey on Deep Learning for Skin Lesion Segmentation](https://mlsanity.com/paper/2206.00356/) (2022)
- [Label fusion and training methods for reliable representation of inter-rater uncertainty](https://mlsanity.com/paper/2202.07550/) (2022)
- [Modeling Annotator Preference and Stochastic Annotation Error for Medical Image Segmentation](https://mlsanity.com/paper/2111.13410/) (2021)
- [Learning to Complement with Multiple Humans](https://mlsanity.com/paper/2311.13172/) (2023)
- [Comprehensive framework for evaluation of deep neural networks in detection and quantification of lymphoma from PET/CT images: clinical insights, pitfalls, and observer agreement analyses](https://mlsanity.com/paper/2311.09614/) (2023)
- [A Novel Surrogate-assisted Evolutionary Algorithm Applied to Partition-based Ensemble Learning](https://mlsanity.com/paper/2104.08048/) (2021)
- [Segmentation Style Discovery: Application to Skin Lesion Images](https://mlsanity.com/paper/2408.02787/) (2024)
- [What Can We Learn from Inter-Annotator Variability in Skin Lesion Segmentation?](https://mlsanity.com/paper/2508.09381/) (2025)
- [A Unified Evaluation Framework for Multi-Annotator Tendency Learning](https://mlsanity.com/paper/2508.10393/) (2025)
- [Deep Learning Techniques for Visual Counting](https://mlsanity.com/paper/2206.03033/) (2022)
- [MSE-Nets: Multi-annotated Semi-supervised Ensemble Networks for Improving Segmentation of Medical Image with Ambiguous Boundaries](https://mlsanity.com/paper/2311.10380/) (2023)
- [Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model](https://mlsanity.com/paper/2111.11055/) (2021)
- [HealNet -- Self-Supervised Acute Wound Heal-Stage Classification](https://mlsanity.com/paper/2206.10536/) (2022)
- [MLV2-Net: Rater-Based Majority-Label Voting for Consistent Meningeal Lymphatic Vessel Segmentation](https://mlsanity.com/paper/2411.08537/) (2024)
- [Leveraging Multi-Rater Annotations to Calibrate Object Detectors in Microscopy Imaging](https://mlsanity.com/paper/2601.23007/) (2026)
- [Harmonized Feature Conditioning and Frequency-Prompt Personalization for Multi-Rater Medical Segmentation](https://mlsanity.com/paper/2605.08210/) (2026)

## Abstract (arXiv)

Medical image segmentation annotations suffer from inter- and intra-observer variations even among experts due to intrinsic differences in human annotators and ambiguous boundaries. Leveraging a collection of annotators' opinions for an image is an interesting way of estimating a gold standard. Although training deep models in a supervised setting with a single annotation per image has been extensively studied, generalizing their training to work with datasets containing multiple annotations per image remains a fairly unexplored problem. In this paper, we propose an approach to handle annotators' disagreements when training a deep model. To this end, we propose an ensemble of Bayesian fully convolutional networks (FCNs) for the segmentation task by considering two major factors in the aggregation of multiple ground truth annotations: (1) handling contradictory annotations in the training data originating from inter-annotator disagreements and (2) improving confidence calibration through the fusion of base models' predictions. We demonstrate the superior performance of our approach on the ISIC Archive and explore the generalization performance of our proposed method by cross-dataset evaluation on the PH2 and DermoFit datasets.