# Methods and open-source toolkit for analyzing and visualizing challenge results

arXiv: https://arxiv.org/abs/1910.05121 · HTML: https://mlsanity.com/paper/1910.05121/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2019.

## Built on by (most cited first, 30 of 60)

- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CT](https://mlsanity.com/paper/2307.01984/) (2023)
- [Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark](https://mlsanity.com/paper/2109.14956/) (2021)
- [CrossMoDA 2021 challenge: Benchmark of Cross-Modality Domain Adaptation techniques for Vestibular Schwannoma and Cochlea Segmentation](https://mlsanity.com/paper/2201.02831/) (2022)
- [Understanding metric-related pitfalls in image analysis validation](https://mlsanity.com/paper/2302.01790/) (2023)
- [Unleashing the Strengths of Unlabeled Data in Pan-cancer Abdominal Organ Quantification: the FLARE22 Challenge](https://mlsanity.com/paper/2308.05862/) (2023)
- [The Multi-modality Cell Segmentation Challenge: Towards Universal Solutions](https://mlsanity.com/paper/2308.05864/) (2023)
- [Overcoming Common Flaws in the Evaluation of Selective Classification Systems](https://mlsanity.com/paper/2407.01032/) (2024)
- [Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?](https://mlsanity.com/paper/2411.03670/) (2024)
- [MIcro-Surgical Anastomose Workflow recognition challenge report](https://mlsanity.com/paper/2103.13111/) (2021)
- [Efficient automatic segmentation for multi-level pulmonary arteries: The PARSE challenge](https://mlsanity.com/paper/2304.03708/) (2023)
- [Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge](https://mlsanity.com/paper/2202.12031/) (2022)
- [Scaling up self-supervised learning for improved surgical foundation models](https://mlsanity.com/paper/2501.09436/) (2025)
- [CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detection](https://mlsanity.com/paper/2302.06294/) (2023)
- [Automatic Organ and Pan-cancer Segmentation in Abdomen CT: the FLARE 2023 Challenge](https://mlsanity.com/paper/2408.12534/) (2024)
- [Why is the winner the best?](https://mlsanity.com/paper/2303.17719/) (2023)
- [Fetal Brain Tissue Annotation and Segmentation Challenge Results](https://mlsanity.com/paper/2204.09573/) (2022)
- [Real-time Instance Segmentation of Surgical Instruments using Attention and Multi-scale Feature Fusion](https://mlsanity.com/paper/2111.04911/) (2021)
- [An objective comparison of methods for augmented reality in laparoscopic liver resection by preoperative-to-intraoperative image fusion](https://mlsanity.com/paper/2401.15753/) (2024)
- [Where is VALDO? VAscular Lesions Detection and segmentatiOn challenge at MICCAI 2021](https://mlsanity.com/paper/2208.07167/) (2022)
- [Deep Rib Fracture Instance Segmentation and Classification from CT on the RibFrac Challenge](https://mlsanity.com/paper/2402.09372/) (2024)
- [Confidence intervals uncovered: Are we ready for real-world medical imaging AI?](https://mlsanity.com/paper/2409.17763/) (2024)
- [The Brain Tumor Segmentation (BraTS) Challenge: Local Synthesis of Healthy Brain Tissue via Inpainting](https://mlsanity.com/paper/2305.08992/) (2023)
- [A Robust Ensemble Algorithm for Ischemic Stroke Lesion Segmentation: Generalizability and Clinical Utility Beyond the ISLES Challenge](https://mlsanity.com/paper/2403.19425/) (2024)
- [PSFHS Challenge Report: Pubic Symphysis and Fetal Head Segmentation from Intrapartum Ultrasound Images](https://mlsanity.com/paper/2409.10980/) (2024)
- [Multi-Class Segmentation of Aortic Branches and Zones in Computed Tomography Angiography: The AortaSeg24 Challenge](https://mlsanity.com/paper/2502.05330/) (2025)
- [Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challenge](https://mlsanity.com/paper/2507.16559/) (2025)
- [NeuCo-Bench: A Novel Benchmark Framework for Neural Embeddings in Earth Observation](https://mlsanity.com/paper/2510.17914/) (2025)
- [Assessing YOLACT++ for real time and robust instance segmentation of medical instruments in endoscopic procedures](https://mlsanity.com/paper/2103.15997/) (2021)

## Abstract (arXiv)

Biomedical challenges have become the de facto standard for benchmarking biomedical image analysis algorithms. While the number of challenges is steadily increasing, surprisingly little effort has been invested in ensuring high quality design, execution and reporting for these international competitions. Specifically, results analysis and visualization in the event of uncertainties have been given almost no attention in the literature. Given these shortcomings, the contribution of this paper is two-fold: (1) We present a set of methods to comprehensively analyze and visualize the results of single-task and multi-task challenges and apply them to a number of simulated and real-life challenges to demonstrate their specific strengths and weaknesses; (2) We release the open-source framework challengeR as part of this work to enable fast and wide adoption of the methodology proposed in this paper. Our approach offers an intuitive way to gain important insights into the relative and absolute performance of algorithms, which cannot be revealed by commonly applied visualization techniques. This is demonstrated by the experiments performed within this work. Our framework could thus become an important tool for analyzing and visualizing challenge results in the field of biomedical image analysis and beyond.