# Towards Ground-truth-free Evaluation of Any Segmentation in Medical Images

arXiv: https://arxiv.org/abs/2409.14874 · HTML: https://mlsanity.com/paper/2409.14874/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Builds on (most cited first, 15 of 15)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Large-Vocabulary Segmentation for Medical Images with Text Prompts](https://mlsanity.com/paper/2312.17183/) (2023)
- [Understanding metric-related pitfalls in image analysis validation](https://mlsanity.com/paper/2302.01790/) (2023)
- [SA-Med2D-20M Dataset: Segment Anything in 2D Medical Imaging with 20 Million masks](https://mlsanity.com/paper/2311.11969/) (2023)
- [Prediction Error Meta Classification in Semantic Segmentation: Detection via Aggregated Dispersion Measures of Softmax Probabilities](https://mlsanity.com/paper/1811.00648/) (2018)
- [Leveraging Uncertainty Estimates for Predicting Segmentation Quality](https://mlsanity.com/paper/1807.00502/) (2018)
- [Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth](https://mlsanity.com/paper/1702.03407/) (2017)
- [Real-time Prediction of Segmentation Quality](https://mlsanity.com/paper/1806.06244/) (2018)
- [Deep Generative Model-based Quality Control for Cardiac MRI Segmentation](https://mlsanity.com/paper/2006.13379/) (2020)
- [FSNet: A Failure Detection Framework for Semantic Segmentation](https://mlsanity.com/paper/2108.08748/) (2021)
- [Conformal Performance Range Prediction for Segmentation Output Quality Control](https://mlsanity.com/paper/2407.13307/) (2024)
- [When is a Foundation Model a Foundation Model](https://mlsanity.com/paper/2309.11510/) (2023)
- [Quality Sentinel: Estimating Label Quality and Errors in Medical Segmentation Datasets](https://mlsanity.com/paper/2406.00327/) (2024)

## Built on by (most cited first, 3 of 3)

- [Metrics that matter: Evaluating image quality metrics for medical image generation](https://mlsanity.com/paper/2505.07175/) (2025)
- [In search of truth: Evaluating concordance of AI-based anatomy segmentation models](https://mlsanity.com/paper/2512.15921/) (2025)
- [Quality-Guided Semi-Supervised Learning for Medical Image Segmentation](https://mlsanity.com/paper/2606.01753/) (2026)

## Abstract (arXiv)

We explore the feasibility and potential of building a ground-truth-free evaluation model to assess the quality of segmentations generated by the Segment Anything Model (SAM) and its variants in medical imaging. This evaluation model estimates segmentation quality scores by analyzing the coherence and consistency between the input images and their corresponding segmentation predictions. Based on prior research, we frame the task of training this model as a regression problem within a supervised learning framework, using Dice scores (and optionally other metrics) along with mean squared error to compute the training loss. The model is trained utilizing a large collection of public datasets of medical images with segmentation predictions from SAM and its variants. We name this model EvanySeg (Evaluation of Any Segmentation in Medical Images). Our exploration of convolution-based models (e.g., ResNet) and transformer-based models (e.g., ViT) suggested that ViT yields better performance for this task. EvanySeg can be employed for various tasks, including: (1) identifying poorly segmented samples by detecting low-percentile segmentation quality scores; (2) benchmarking segmentation models without ground truth by averaging quality scores across test samples; (3) alerting human experts to poor-quality segmentation predictions during human-AI collaboration by applying a threshold within the score space; and (4) selecting the best segmentation prediction for each test sample at test time when multiple segmentation models are available, by choosing the prediction with the highest quality score. Models and code will be made available at https://github.com/ahjolsenbics/EvanySeg.