mlsanityData to 7 Oct 2026

Understanding metric-related pitfalls in image analysis validation

arXiv 2302.01790 ↗First posted 3 Feb 2023cs.CVcited by 57

Not on the map. Why?

Abstract (arXiv)

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 29 of its 108 references that are in the map. First posted on arXiv 3 Feb 2023.

Builds on

29 of 108 references.

References · 29

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 504
  4. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  5. 2019Verified Uncertainty Calibrationcited by 307
  6. 2020clDice -- A Novel Topology-Preserving Loss Function for Tubular Structure Segmentationcited by 286
  7. 2022Metrics reloaded: Recommendations for image analysis validationcited by 190
  8. 2021Boundary IoU: Improving Object-Centric Image Segmentation Evaluationcited by 186
  9. 2018Cell Detection with Star-convex Polygonscited by 182
  10. 2019Evaluating model calibration in classificationcited by 165
  11. 2020Metrics for Multi-Class Classification: an Overviewcited by 140
  12. 2019Deep Semantic Segmentation of Natural and Medical Images: A Reviewcited by 124
17 more
  1. 2017Semantic Instance Segmentation with a Discriminative Loss Functioncited by 118
  2. 2018Why rankings of biomedical image analysis competitions should be interpreted with carecited by 90
  3. 2019SSAP: Single-Shot Instance Segmentation With Affinity Pyramidcited by 85
  4. 2019Calibration tests in multi-class classification: A unifying frameworkcited by 79
  5. 2016Deep Watershed Transform for Instance Segmentationcited by 77
  6. 2022Better Uncertainty Calibration via Proper Scores for Classification and Beyondcited by 65
  7. 2019Methods and open-source toolkit for analyzing and visualizing challenge resultscited by 60
  8. 2021Common Limitations of Image Processing Metrics: A Picture Storycited by 58
  9. 2022A Consistent and Differentiable Lp Canonical Calibration Error Estimatorcited by 44
  10. 2017Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathycited by 42
  11. 2021Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficientcited by 26
  12. 2022Analysis and Comparison of Classification Metricscited by 16
  13. 2019Instance Segmentation of Biological Images Using Harmonic Embeddingscited by 14
  14. 2020PatchPerPix for Instance Segmentationcited by 8
  15. 2020The MCC-F1 curve: a performance evaluation technique for binary classificationcited by 6
  16. 2021Anatomical and Diagnostic Bayesian Segmentation in Prostate MRI -Should Different Clinical Objectives Mandate Different Loss Functions?cited by 4
  17. 2022Sources of performance variability in deep learning-based polyp detectioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 18; 2025: 10; 2026: 18.

57 papers.

Sort:

By area:

  1. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  2. 2023The TopCoW Challenge -- Topology-Aware Circle of Willis Segmentation for CT and MR Angiographycited by 48
  3. 2024nnMamba: 3D Biomedical Image Segmentation, Classification and Landmark Detection with State Space Modelcited by 45
  4. 2024Centerline Boundary Dice Loss for Vascular Segmentationcited by 27
  5. 2023Deep Interactive Segmentation of Medical Images: A Systematic Review and Taxonomycited by 26
  6. 2024Similarity and Quality Metrics for MR Image-To-Image Translationcited by 20
  7. 2024Fr'echet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasetscited by 19
  8. 2024DeNVeR: Deformable Neural Vessel Representations for Unsupervised Video Vessel Segmentationcited by 13
  9. 2023Why is plausibility surprisingly problematic as an XAI criterion?cited by 9
  10. 2023Multi-Head Multi-Loss Model Calibrationcited by 8
  11. 2024A Robust Ensemble Algorithm for Ischemic Stroke Lesion Segmentation: Generalizability and Clinical Utility Beyond the ISLES Challengecited by 8
  12. 2024Every Component Counts: Rethinking the Measure of Success for Medical Semantic Segmentation in Multi-Instance Segmentation Taskscited by 8
38 more
  1. 2024BraTS-Path Challenge: Assessing Heterogeneous Histopathologic Brain Tumor Sub-regionscited by 6
  2. 2024Quality Assured: Rethinking Annotation Strategies in Imaging AIcited by 6
  3. 2024Diffuse-UDA: Addressing Unsupervised Domain Adaptation in Medical Image Segmentation with Appearance and Structure Aligned Diffusion Modelscited by 6
  4. 2025Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challengecited by 6
  5. 2026TopoLoRA-SAM: Topology-Aware Parameter-Efficient Adaptation of Foundation Segmenters for Thin-Structure and Cross-Domain Binary Semantic Segmentationcited by 6
  6. 2023Performance Metrics for Probabilistic Ordinal Classifierscited by 5
  7. 2023Pearson-Matthews correlation coefficients for binary and multinary classification and hypothesis testingcited by 4
  8. 2024Five Pitfalls When Assessing Synthetic Medical Images with Reference Metricscited by 4
  9. 2024Towards Ground-truth-free Evaluation of Any Segmentation in Medical Imagescited by 3
  10. 2024Understanding implementation pitfalls of distance-based metrics for image segmentationcited by 3
  11. 2024Strategies to Improve Real-World Applicability of Laparoscopic Anatomy Segmentation Modelscited by 2
  12. 2024Open Challenges on Fairness of Artificial Intelligence in Medical Imaging Applicationscited by 2
  13. 2024Beyond Knowledge Silos: Task Fingerprinting for Democratization of Medical Imaging AIcited by 2
  14. 2025The MICCAI Federated Tumor Segmentation (FeTS) Challenge 2024: Efficient and Robust Aggregation Methods for Federated Learningcited by 2
  15. 2025Auditing Significance, Metric Choice, and Demographic Fairness in Medical AI Challengescited by 2
  16. 2023Confidence Intervals for Performance Estimates in Brain MRI Segmentationcited by 1
  17. 2024Costal Cartilage Segmentation with Topology Guided Deformable Mamba: Method and Benchmarkcited by 1
  18. 2025Benchmarking and Evaluation of AI Models in Biology: Outcomes and Recommendations from the CZI Virtual Cells Workshopcited by 1
  19. 2026TriALS: Triphasic-Aided Liver Lesion Segmentation Benchmark in Non-Contrast CTcited by 1
  20. 2023Cross-Validation Is All You Need: A Statistical Approach To Label Noise Estimationcited by 0
  21. 2023SHAMSUL: Systematic Holistic Analysis to investigate Medical Significance Utilizing Local interpretability methods in deep learning for chest radiography pathology predictioncited by 0
  22. 2023Application-driven Validation of Posteriors in Inverse Problemscited by 0
  23. 2024Automated neuroradiological support systems for multiple cerebrovascular disease markers -- A systematic review and meta-analysiscited by 0
  24. 2025A Novel Distance-Based Metric for Quality Assessment in Image Segmentationcited by 0
  25. 2025A Multi-Resolution Benchmark Framework for Spatial Reasoning Assessment in Neural Networkscited by 0
  26. 2025Federative ischemic stroke segmentation as alternative to overcome domain-shift multi-institution challengescited by 0
  27. 2025MeshMetrics: A Precise Implementation of Distance-Based Image Segmentation Metricscited by 0
  28. 2025Clinical Uncertainty Impacts Machine Learning Evaluationscited by 0
  29. 2025Weighted MCC: A Robust Measure of Multiclass Classifier Performance for Observations with Individual Weightscited by 0
  30. 2026Pareto-Guided Optimization for Uncertainty-Aware Medical Image Segmentationcited by 0
  31. 2026Multiclass Calibration Assessment and Recalibration of Probability Predictions via the Linear Log Odds Calibration Functioncited by 0
  32. 2026Brain Tumor Segmentation with Special Emphasis on the Non-Enhancing Brain Tumor Compartmentcited by 0
  33. 2026NucEval: A Robust Evaluation Framework for Nuclear Instance Segmentationcited by 0
  34. 2026Deep Slice Interpolation for Reducing Through-Plane Anisotropy and Noise in Head CTcited by 0
  35. 2026Metrics or Mirage? An Audit of Evaluation Inconsistencies in Colonoscopy Polyp Segmentation Benchmarkscited by 0
  36. 2026OvAi Focus: AI-based Multi-class Segmentation of Functional Ovaries and Adnexal Masses in Gynecological Ultrasoundcited by 0
  37. 2026Toward Better Assessment of LLMs' Performance in Clinical Error Detectioncited by 0
  38. 2026Beyond Accuracy: Quantifying Pulmonary Attribution in Anatomy-Guided Chest X-Ray Classification Under Domain Shiftcited by 0

50 most cited of 57 All 57 (JSON)