# Common Limitations of Image Processing Metrics: A Picture Story arXiv: https://arxiv.org/abs/2104.05642 ยท HTML: https://mlsanity.com/paper/2104.05642/ ## Key facts - Cited by 59 papers in the map (data through 14 Oct 2026). - Builds on 12 of its 59 references that are in the map. - First posted on arXiv 12 Apr 2021. ## Builds on (most cited first, 12 of 12) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018) - [Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations](https://mlsanity.com/paper/1707.03237/) (2017) - [clDice -- A Novel Topology-Preserving Loss Function for Tubular Structure Segmentation](https://mlsanity.com/paper/2003.07311/) (2020) - [Boundary IoU: Improving Object-Centric Image Segmentation Evaluation](https://mlsanity.com/paper/2103.16562/) (2021) - [Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation](https://mlsanity.com/paper/1808.01200/) (2018) - [Metrics for Multi-Class Classification: an Overview](https://mlsanity.com/paper/2008.05756/) (2020) - [Deep Semantic Segmentation of Natural and Medical Images: A Review](https://mlsanity.com/paper/1910.07655/) (2019) - [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018) - [Retina U-Net: Embarrassingly Simple Exploitation of Segmentation Supervision for Medical Object Detection](https://mlsanity.com/paper/1811.08661/) (2018) - [Localization Recall Precision (LRP): A New Performance Metric for Object Detection](https://mlsanity.com/paper/1807.01696/) (2018) - [Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficient](https://mlsanity.com/paper/2103.06205/) (2021) ## Built on by (most cited first, 30 of 59) - [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021) - [MONAI: An open-source framework for deep learning in healthcare](https://mlsanity.com/paper/2211.02701/) (2022) - [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022) - [UNETR++: Delving into Efficient and Accurate 3D Medical Image Segmentation](https://mlsanity.com/paper/2212.04497/) (2022) - [MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation](https://mlsanity.com/paper/2110.01406/) (2021) - [Understanding metric-related pitfalls in image analysis validation](https://mlsanity.com/paper/2302.01790/) (2023) - [A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification](https://mlsanity.com/paper/2211.15259/) (2022) - [QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results](https://mlsanity.com/paper/2112.10074/) (2021) - [Improving anatomical plausibility in medical image segmentation via hybrid graph neural networks: applications to chest x-ray analysis](https://mlsanity.com/paper/2203.10977/) (2022) - [Shifts 2.0: Extending The Dataset of Real Distributional Shifts](https://mlsanity.com/paper/2206.15407/) (2022) - [Why is the winner the best?](https://mlsanity.com/paper/2303.17719/) (2023) - [Labeling instructions matter in biomedical image analysis](https://mlsanity.com/paper/2207.09899/) (2022) - [Medical Image Registration via Neural Fields](https://mlsanity.com/paper/2206.03111/) (2022) - [The Open Kidney Ultrasound Data Set](https://mlsanity.com/paper/2206.06657/) (2022) - [Calibrating the Dice loss to handle neural network overconfidence for biomedical image segmentation](https://mlsanity.com/paper/2111.00528/) (2021) - [ProstAttention-Net: A deep attention model for prostate cancer segmentation by aggressiveness in MRI scans](https://mlsanity.com/paper/2211.13238/) (2022) - [Panoptica -- instance-wise evaluation of 3D semantic and instance segmentation maps](https://mlsanity.com/paper/2312.02608/) (2023) - [Preoperative brain tumor imaging: models and software for segmentation and standardized reporting](https://mlsanity.com/paper/2204.14199/) (2022) - [Why is plausibility surprisingly problematic as an XAI criterion?](https://mlsanity.com/paper/2303.17707/) (2023) - [Approaching Peak Ground Truth](https://mlsanity.com/paper/2301.00243/) (2022) - [Tackling Bias in the Dice Similarity Coefficient: Introducing nDSC for White Matter Lesion Segmentation](https://mlsanity.com/paper/2302.05432/) (2023) - [Rethinking Machine Learning Model Evaluation in Pathology](https://mlsanity.com/paper/2204.05205/) (2022) - [Semantic segmentation of multispectral photoacoustic images using deep learning](https://mlsanity.com/paper/2105.09624/) (2021) - [Improving Segmentation of Objects with Varying Sizes in Biomedical Images using Instance-wise and Center-of-Instance Segmentation Loss Function](https://mlsanity.com/paper/2304.06229/) (2023) - [Comparison of Evaluation Metrics for Landmark Detection in CMR Images](https://mlsanity.com/paper/2201.10410/) (2022) - [Are Face Detection Models Biased?](https://mlsanity.com/paper/2211.03588/) (2022) - [A new dataset for measuring the performance of blood vessel segmentation methods under distribution shifts](https://mlsanity.com/paper/2301.04517/) (2023) - [Sources of performance variability in deep learning-based polyp detection](https://mlsanity.com/paper/2211.09708/) (2022) - [Revisiting Hidden Representations in Transfer Learning for Medical Imaging](https://mlsanity.com/paper/2302.08272/) (2023) - [Architecture Analysis and Benchmarking of 3D U-shaped Deep Learning Models for Thoracic Anatomical Segmentation](https://mlsanity.com/paper/2402.03230/) (2024) ## Abstract (arXiv) While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.