# Metrics or Mirage? An Audit of Evaluation Inconsistencies in Colonoscopy Polyp Segmentation Benchmarks

arXiv: https://arxiv.org/abs/2607.08203 · HTML: https://mlsanity.com/paper/2607.08203/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 9 Jul 2026.

## Builds on (most cited first, 20 of 20)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [PraNet: Parallel Reverse Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2006.11392/) (2020)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images](https://mlsanity.com/paper/1612.00799/) (2016)
- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers](https://mlsanity.com/paper/2108.06932/) (2021)
- [Shallow Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2108.00882/) (2021)
- [Context-aware Cross-level Fusion Network for Camouflaged Object Detection](https://mlsanity.com/paper/2105.12555/) (2021)
- [Video Polyp Segmentation: A Deep Learning Perspective](https://mlsanity.com/paper/2203.14291/) (2022)
- [A multi-centre polyp detection and segmentation dataset for generalisability assessment](https://mlsanity.com/paper/2106.04463/) (2021)
- [UACANet: Uncertainty Augmented Context Attention for Polyp Segmentation](https://mlsanity.com/paper/2107.02368/) (2021)
- [Understanding metric-related pitfalls in image analysis validation](https://mlsanity.com/paper/2302.01790/) (2023)
- [CaraNet: Context Axial Reverse Attention Network for Segmentation of Small Medical Objects](https://mlsanity.com/paper/2301.13366/) (2023)
- [ColonFormer: An Efficient Transformer based Method for Colon Polyp Segmentation](https://mlsanity.com/paper/2205.08473/) (2022)
- [TransNetR: Transformer-based Residual Network for Polyp Segmentation with Multi-Center Out-of-Distribution Testing](https://mlsanity.com/paper/2303.07428/) (2023)
- [DuAT: Dual-Aggregation Transformer Network for Medical Image Segmentation](https://mlsanity.com/paper/2212.11677/) (2022)
- [G-CASCADE: Efficient Cascaded Graph Convolutional Decoding for 2D Medical Image Segmentation](https://mlsanity.com/paper/2310.16175/) (2023)
- [A Survey on Deep Learning for Polyp Segmentation: Techniques, Challenges and Future Trends](https://mlsanity.com/paper/2311.18373/) (2023)
- [ESFPNet: efficient deep learning architecture for real-time lesion segmentation in autofluorescence bronchoscopic video](https://mlsanity.com/paper/2207.07759/) (2022)

## Abstract (arXiv)

Progress in colonoscopy polyp segmentation is routinely reported through leaderboard comparisons on a small set of public benchmarks. We argue that this apparent progress is difficult to verify: a systematic audit of 27 papers published between 2015 and 2026 reveals three structural problems in how the community evaluates models. First, 25 of 27 papers textitomit the Hausdorff distance. Hausdorff distance is a boundary-accuracy metric with direct clinical relevance for detecting flat or small polyps, and is a standard in radiotherapy segmentation. Second, at least five textitincompatible train/test split protocols co-exist across papers reporting results on the same two datasets (Kvasir-SEG and CVC-ClinicDB), making published Dice scores non-comparable even when they appear in the same leaderboard column. Third, 26 of 27 papers make textitperformance claims without any statistical significance test. Strikingly, four papers published after the Metrics Reloaded framework~citemetricsreloaded2024 (Maier-Hein et al., textitNature Methods 2024) perpetuate these same problems, suggesting that general-purpose metric guidance has not yet reached the colonoscopy sub-community. To show these problems are not merely cosmetic, we re-evaluate five representative models under three controlled protocols with a single uniform scorer, and find that the reported metric conceals large boundary and recall failures, that the ``best'' model changes with the metric, and that near-tied rankings reverse across random splits. We propose a five-point Polyp Segmentation Reporting Checklist~(PSRC) as a lightweight, domain-adapted corrective.