# Optimization for Medical Image Segmentation: Theory and Practice when evaluating with Dice Score or Jaccard Index

arXiv: https://arxiv.org/abs/2010.13499 · HTML: https://mlsanity.com/paper/2010.13499/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2020.

## Built on by (most cited first, 30 of 52)

- [blob loss: instance imbalance aware loss functions for semantic segmentation](https://mlsanity.com/paper/2205.08209/) (2022)
- [Calibrating the Dice loss to handle neural network overconfidence for biomedical image segmentation](https://mlsanity.com/paper/2111.00528/) (2021)
- [Do We Really Need Dice? The Hidden Region-Size Biases of Segmentation Losses](https://mlsanity.com/paper/2104.08717/) (2021)
- [Detect-and-Segment: a Deep Learning Approach to Automate Wound Image Segmentation](https://mlsanity.com/paper/2111.01590/) (2021)
- [Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labels](https://mlsanity.com/paper/2302.05666/) (2023)
- [On the relationship between calibrated predictors and unbiased volume estimation](https://mlsanity.com/paper/2112.12560/) (2021)
- [On the Optimal Combination of Cross-Entropy and Soft Dice Losses for Lesion Segmentation with Out-of-Distribution Robustness](https://mlsanity.com/paper/2209.06078/) (2022)
- [TATL: Task Agnostic Transfer Learning for Skin Attributes Detection](https://mlsanity.com/paper/2104.01641/) (2021)
- [EllSeg-Gen, towards Domain Generalization for head-mounted eyetracking](https://mlsanity.com/paper/2205.01947/) (2022)
- [Theoretical analysis and experimental validation of volume bias of soft Dice optimized segmentation maps in the context of inherent uncertainty](https://mlsanity.com/paper/2211.04161/) (2022)
- [ViTBIS: Vision Transformer for Biomedical Image Segmentation](https://mlsanity.com/paper/2201.05920/) (2022)
- [The Dice loss in the context of missing or empty labels: Introducing Phi and ε](https://mlsanity.com/paper/2207.09521/) (2022)
- [Dice Semimetric Losses: Optimizing the Dice Score with Soft Labels](https://mlsanity.com/paper/2303.16296/) (2023)
- [A new baseline for retinal vessel segmentation: Numerical identification and correction of methodological inconsistencies affecting 100+ papers](https://mlsanity.com/paper/2111.03853/) (2021)
- [Distribution-aware Margin Calibration for Semantic Segmentation in Images](https://mlsanity.com/paper/2112.11554/) (2021)
- [DGNet: Distribution Guided Efficient Learning for Oil Spill Image Segmentation](https://mlsanity.com/paper/2301.01202/) (2022)
- [Advancing Eosinophilic Esophagitis Diagnosis and Phenotype Assessment with Deep Learning Computer Vision](https://mlsanity.com/paper/2101.05326/) (2021)
- [Deep Learning Analysis of Cardiac MRI in Legacy Datasets: Multi-Ethnic Study of Atherosclerosis](https://mlsanity.com/paper/2110.15144/) (2021)
- [AGSENet: A Robust Road Ponding Detection Method for Proactive Traffic Safety](https://mlsanity.com/paper/2410.16999/) (2024)
- [ReCoSeg++:Extended Residual-Guided Cross-Modal Diffusion for Brain Tumor Segmentation](https://mlsanity.com/paper/2508.01058/) (2025)
- [MRF-UNets: Searching UNet with Markov Random Fields](https://mlsanity.com/paper/2207.06168/) (2022)
- [Brain Tumor Segmentation from MRI Images using Deep Learning Techniques](https://mlsanity.com/paper/2305.00257/) (2023)
- [Integrating Deep Learning in Cardiology: A Comprehensive Review of Atrial Fibrillation, Left Atrial Scar Segmentation, and the Frontiers of State-of-the-Art Techniques](https://mlsanity.com/paper/2407.09561/) (2024)
- [GBT-SAM: A Parameter-Efficient Depth-Aware Model for Generalizable Brain tumour Segmentation on mp-MRI](https://mlsanity.com/paper/2503.04325/) (2025)
- [Noisy Image Segmentation With Soft-Dice](https://mlsanity.com/paper/2304.00801/) (2023)
- [A Review of Benchmarks for Visual Defect Detection in the Manufacturing Industry](https://mlsanity.com/paper/2305.13261/) (2023)
- [CNN-based explanation ensembling for dataset, representation and explanations evaluation](https://mlsanity.com/paper/2404.10387/) (2024)
- [RankSEG-RMA: An Efficient Segmentation Algorithm via Reciprocal Moment Approximation](https://mlsanity.com/paper/2510.15362/) (2025)
- [A new smart-cropping pipeline for prostate segmentation using deep learning networks](https://mlsanity.com/paper/2107.02476/) (2021)
- [Automated risk classification of colon biopsies based on semantic segmentation of histopathology images](https://mlsanity.com/paper/2109.07892/) (2021)

## Abstract (arXiv)

In many medical imaging and classical computer vision tasks, the Dice score and Jaccard index are used to evaluate the segmentation performance. Despite the existence and great empirical success of metric-sensitive losses, i.e. relaxations of these metrics such as soft Dice, soft Jaccard and Lovasz-Softmax, many researchers still use per-pixel losses, such as (weighted) cross-entropy to train CNNs for segmentation. Therefore, the target metric is in many cases not directly optimized. We investigate from a theoretical perspective, the relation within the group of metric-sensitive loss functions and question the existence of an optimal weighting scheme for weighted cross-entropy to optimize the Dice score and Jaccard index at test time. We find that the Dice score and Jaccard index approximate each other relatively and absolutely, but we find no such approximation for a weighted Hamming similarity. For the Tversky loss, the approximation gets monotonically worse when deviating from the trivial weight setting where soft Tversky equals soft Dice. We verify these results empirically in an extensive validation on six medical segmentation tasks and can confirm that metric-sensitive losses are superior to cross-entropy based loss functions in case of evaluation with Dice Score or Jaccard Index. This further holds in a multi-class setting, and across different object sizes and foreground/background ratios. These results encourage a wider adoption of metric-sensitive loss functions for medical segmentation tasks where the performance measure of interest is the Dice score or Jaccard index.