# Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficient

arXiv: https://arxiv.org/abs/2103.06205 · HTML: https://mlsanity.com/paper/2103.06205/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 75 references that are in the map.
- First posted on arXiv 10 Mar 2021.

## Builds on (most cited first, 18 of 18)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations](https://mlsanity.com/paper/1707.03237/) (2017)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation](https://mlsanity.com/paper/1603.05959/) (2016)
- [Tversky loss function for image segmentation using 3D fully convolutional deep networks](https://mlsanity.com/paper/1706.05721/) (2017)
- [The Importance of Skip Connections in Biomedical Image Segmentation](https://mlsanity.com/paper/1608.04117/) (2016)
- [Reducing the Hausdorff Distance in Medical Image Segmentation with Convolutional Neural Networks](https://mlsanity.com/paper/1904.10030/) (2019)
- [nnU-Net for Brain Tumor Segmentation](https://mlsanity.com/paper/2011.00848/) (2020)
- [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018)
- [Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy](https://mlsanity.com/paper/1809.04430/) (2018)
- [No New-Net](https://mlsanity.com/paper/1809.10483/) (2018)
- [Brain Tumor Segmentation using an Ensemble of 3D U-Nets and Overall Survival Prediction using Radiomic Features](https://mlsanity.com/paper/1812.01049/) (2018)
- [Asymmetric Loss Functions and Deep Densely Connected Networks for Highly Imbalanced Medical Image Segmentation: Application to Multiple Sclerosis Lesion Detection](https://mlsanity.com/paper/1803.11078/) (2018)
- [pymia: A Python package for data handling and evaluation in deep learning-based medical image analysis](https://mlsanity.com/paper/2010.03639/) (2020)
- [Weakly-Supervised White and Grey Matter Segmentation in 3D Brain Ultrasound](https://mlsanity.com/paper/1904.05191/) (2019)

## Built on by (most cited first, 26 of 26)

- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [Common Limitations of Image Processing Metrics: A Picture Story](https://mlsanity.com/paper/2104.05642/) (2021)
- [Understanding metric-related pitfalls in image analysis validation](https://mlsanity.com/paper/2302.01790/) (2023)
- [Why is the winner the best?](https://mlsanity.com/paper/2303.17719/) (2023)
- [blob loss: instance imbalance aware loss functions for semantic segmentation](https://mlsanity.com/paper/2205.08209/) (2022)
- [QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge](https://mlsanity.com/paper/2405.18435/) (2024)
- [Panoptica -- instance-wise evaluation of 3D semantic and instance segmentation maps](https://mlsanity.com/paper/2312.02608/) (2023)
- [A Dempster-Shafer approach to trustworthy AI with application to fetal brain MRI segmentation](https://mlsanity.com/paper/2204.02779/) (2022)
- [Why is plausibility surprisingly problematic as an XAI criterion?](https://mlsanity.com/paper/2303.17707/) (2023)
- [Approaching Peak Ground Truth](https://mlsanity.com/paper/2301.00243/) (2022)
- [Tackling Bias in the Dice Similarity Coefficient: Introducing nDSC for White Matter Lesion Segmentation](https://mlsanity.com/paper/2302.05432/) (2023)
- [The Brain Tumor Segmentation (BraTS) Challenge: Local Synthesis of Healthy Brain Tissue via Inpainting](https://mlsanity.com/paper/2305.08992/) (2023)
- [A Robust Ensemble Algorithm for Ischemic Stroke Lesion Segmentation: Generalizability and Clinical Utility Beyond the ISLES Challenge](https://mlsanity.com/paper/2403.19425/) (2024)
- [Deep Quality Estimation: Creating Surrogate Models for Human Quality Ratings](https://mlsanity.com/paper/2205.10355/) (2022)
- [Generalized Wasserstein Dice Loss, Test-time Augmentation, and Transformers for the BraTS 2021 challenge](https://mlsanity.com/paper/2112.13054/) (2021)
- [Sources of performance variability in deep learning-based polyp detection](https://mlsanity.com/paper/2211.09708/) (2022)
- [Strategies to Improve Real-World Applicability of Laparoscopic Anatomy Segmentation Models](https://mlsanity.com/paper/2403.17192/) (2024)
- [MultiOrg: A Multi-rater Organoid-detection Dataset](https://mlsanity.com/paper/2410.14612/) (2024)
- [A Domain-specific Perceptual Metric via Contrastive Self-supervised Representation: Applications on Natural and Medical Images](https://mlsanity.com/paper/2212.01577/) (2022)
- [Primitive Simultaneous Optimization of Similarity Metrics for Image Registration](https://mlsanity.com/paper/2304.01601/) (2023)
- [Framing image registration as a landmark detection problem for label-noise-aware task representation (HitR)](https://mlsanity.com/paper/2308.01318/) (2023)
- [All Sizes Matter: Improving Volumetric Brain Segmentation on Small Lesions](https://mlsanity.com/paper/2310.02829/) (2023)
- [Deep Learning Pipeline for Fully Automated Myocardial Infarct Segmentation from Clinical Cardiac MR Scans](https://mlsanity.com/paper/2502.03272/) (2025)
- [Enhancing efficiency in paediatric brain tumour segmentation using a pathologically diverse single-center clinical dataset](https://mlsanity.com/paper/2507.22152/) (2025)
- [Training the next generation of physicians for artificial intelligence-assisted clinical neuroradiology: ASNR MICCAI Brain Tumor Segmentation (BraTS) 2025 Lighthouse Challenge education platform](https://mlsanity.com/paper/2509.17281/) (2025)
- [Learning from Disagreement: A Group Decision Simulation Framework for Robust Medical Image Segmentation](https://mlsanity.com/paper/2510.10462/) (2025)

## Abstract (arXiv)

Metrics optimized in complex machine learning tasks are often selected in an ad-hoc manner. It is unknown how they align with human expert perception. We explore the correlations between established quantitative segmentation quality metrics and qualitative evaluations by professionally trained human raters. Therefore, we conduct psychophysical experiments for two complex biomedical semantic segmentation problems. We discover that current standard metrics and loss functions correlate only moderately with the segmentation quality assessment of experts. Importantly, this effect is particularly pronounced for clinically relevant structures, such as the enhancing tumor compartment of glioma in brain magnetic resonance and grey matter in ultrasound imaging. It is often unclear how to optimize abstract metrics, such as human expert perception, in convolutional neural network (CNN) training. To cope with this challenge, we propose a novel strategy employing techniques of classical statistics to create complementary compound loss functions to better approximate human expert perception. Across all rating experiments, human experts consistently scored computer-generated segmentations better than the human-curated reference labels. Our results, therefore, strongly question many current practices in medical image segmentation and provide meaningful cues for future research.