mlsanityData to 14 Oct 2026

Common Limitations of Image Processing Metrics: A Picture Story

arXiv 2104.05642 ↗First posted 12 Apr 2021eess.IV cs.CVcited by 59

Not on the map. Why?

Abstract (arXiv)

While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Builds on 12 of its 59 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

12 of 59 references.

References · 12

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2018Panoptic Segmentationcited by 802
  3. 2017Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentationscited by 620
  4. 2020clDice -- A Novel Topology-Preserving Loss Function for Tubular Structure Segmentationcited by 294
  5. 2021Boundary IoU: Improving Object-Centric Image Segmentation Evaluationcited by 188
  6. 2018Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentationcited by 146
  7. 2020Metrics for Multi-Class Classification: an Overviewcited by 143
  8. 2019Deep Semantic Segmentation of Natural and Medical Images: A Reviewcited by 125
  9. 2018Why rankings of biomedical image analysis competitions should be interpreted with carecited by 91
  10. 2018Retina U-Net: Embarrassingly Simple Exploitation of Segmentation Supervision for Medical Object Detectioncited by 47
  11. 2018Localization Recall Precision (LRP): A New Performance Metric for Object Detectioncited by 31
  12. 2021Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficientcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 19; 2023: 11; 2024: 6; 2025: 7; 2026: 9.

59 papers.

Sort:

  1. 2021The Medical Segmentation Decathloncited by 628
  2. 2022MONAI: An open-source framework for deep learning in healthcarecited by 448
  3. 2022Metrics reloaded: Recommendations for image analysis validationcited by 195
  4. 2022UNETR++: Delving into Efficient and Accurate 3D Medical Image Segmentationcited by 112
  5. 2021MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluationcited by 88
  6. 2023Understanding metric-related pitfalls in image analysis validationcited by 58
  7. 2022A Call to Reflect on Evaluation Practices for Failure Detection in Image Classificationcited by 52
  8. 2021QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Resultscited by 28
  9. 2022Improving anatomical plausibility in medical image segmentation via hybrid graph neural networks: applications to chest x-ray analysiscited by 25
  10. 2022Shifts 2.0: Extending The Dataset of Real Distributional Shiftscited by 24
  11. 2023Why is the winner the best?cited by 21
  12. 2022Labeling instructions matter in biomedical image analysiscited by 20
38 more
  1. 2022Medical Image Registration via Neural Fieldscited by 19
  2. 2022The Open Kidney Ultrasound Data Setcited by 18
  3. 2021Calibrating the Dice loss to handle neural network overconfidence for biomedical image segmentationcited by 17
  4. 2022ProstAttention-Net: A deep attention model for prostate cancer segmentation by aggressiveness in MRI scanscited by 14
  5. 2023Panoptica -- instance-wise evaluation of 3D semantic and instance segmentation mapscited by 12
  6. 2022Preoperative brain tumor imaging: models and software for segmentation and standardized reportingcited by 9
  7. 2023Why is plausibility surprisingly problematic as an XAI criterion?cited by 9
  8. 2022Approaching Peak Ground Truthcited by 8
  9. 2023Tackling Bias in the Dice Similarity Coefficient: Introducing nDSC for White Matter Lesion Segmentationcited by 8
  10. 2022Rethinking Machine Learning Model Evaluation in Pathologycited by 6
  11. 2021Semantic segmentation of multispectral photoacoustic images using deep learningcited by 4
  12. 2023Improving Segmentation of Objects with Varying Sizes in Biomedical Images using Instance-wise and Center-of-Instance Segmentation Loss Functioncited by 4
  13. 2022Comparison of Evaluation Metrics for Landmark Detection in CMR Imagescited by 3
  14. 2022Are Face Detection Models Biased?cited by 3
  15. 2023A new dataset for measuring the performance of blood vessel segmentation methods under distribution shiftscited by 3
  16. 2022Sources of performance variability in deep learning-based polyp detectioncited by 2
  17. 2023Revisiting Hidden Representations in Transfer Learning for Medical Imagingcited by 2
  18. 2024Architecture Analysis and Benchmarking of 3D U-shaped Deep Learning Models for Thoracic Anatomical Segmentationcited by 2
  19. 2025Pixel-wise Modulated Dice Loss for Medical Image Segmentationcited by 2
  20. 2026Towards Fairness under Label Bias in Image Segmentation: Impact, Measurement and Mitigationcited by 2
  21. 2024CT evaluation of 2D and 3D holistic deep learning methods for the volumetric segmentation of airway lesionscited by 1
  22. 2024Automatic brain tumor segmentation in 2D intra-operative ultrasound images using magnetic resonance imaging tumor annotationscited by 1
  23. 2025Graph Attention Convolutional U-NET: A Semantic Segmentation Model for Identifying Flooded Areascited by 1
  24. 2025TopoSculpt: Betti-Steered Topological Sculpting of 3D Fine-grained Tubular Shapescited by 1
  25. 2026Mask-HybridGNet: Graph-based segmentation with emergent anatomical correspondence from pixel-level supervisioncited by 1
  26. 2021Chameleon: A Semi-AutoML framework targeting quick and scalable development and deployment of production-ready ML systems for SMEscited by 0
  27. 2021Robust deep learning-based semantic organ segmentation in hyperspectral imagescited by 0
  28. 2022A deep learning framework for the detection and quantification of drusen and reticular pseudodrusen on optical coherence tomographycited by 0
  29. 2022mvHOTA: A multi-view higher order tracking accuracy metric to measure spatial and temporal associations in multi-point detectioncited by 0
  30. 2022From Correlation to Causation: Formalizing Interpretable Machine Learning as a Statistical Processcited by 0
  31. 2023The STOIC2021 COVID-19 AI challenge: applying reusable training methodologies to private datacited by 0
  32. 2023Can ultrasound confidence maps predict sonographers' labeling variability?cited by 0
  33. 2023Glioblastoma Tumor Segmentation using an Ensemble of Vision Transformerscited by 0
  34. 2024CT respiratory motion synthesis using joint supervised and adversarial learningcited by 0
  35. 2024How Much Data are Enough? Investigating Dataset Requirements for Patch-Based Brain MRI Segmentation Taskscited by 0
  36. 2024Evaluation of Blood Vessel Segmentation Methods on Hard-to-Detect Vascular Structurescited by 0
  37. 2025Learning from Anatomy: Supervised Anatomical Pretraining (SAP) for Improved Metastatic Bone Disease Segmentation in Whole-Body MRIcited by 0
  38. 2025Sex-based Bias Inherent in the Dice Similarity Coefficient: A Model Independent Analysis for Multiple Anatomical Structurescited by 0

50 most cited of 59 All 59 (JSON)