Quality Sentinel: Estimating Label Quality and Errors in Medical Segmentation Datasets
Not on the map. Why?
Abstract (arXiv)
An increasing number of public datasets have shown a transformative impact on automated medical segmentation. However, these datasets are often with varying label quality, ranging from manual expert annotations to AI-generated pseudo-annotations. There is no systematic, reliable, and automatic quality control (QC). To fill in this bridge, we introduce a regression model, Quality Sentinel, to estimate label quality compared with manual annotations in medical segmentation datasets. This regression model was trained on over 4 million image-label pairs created by us. Each pair presents a varying but quantified label quality based on manual annotations, which enable us to predict the label quality of any image-label pairs in the inference. Our Quality Sentinel can predict the label quality of 142 body structures. The predicted label quality quantified by Dice Similarity Coefficient (DSC) shares a strong correlation with ground truth quality, with a positive correlation coefficient (r=0.902). Quality Sentinel has found multiple impactful use cases. (I) We evaluated label quality in publicly available datasets, where quality highly varies across different datasets. Our analysis also uncovers that male and younger subjects exhibit significantly higher quality. (II) We identified and corrected poorly annotated labels, achieving 1/3 reduction in annotation costs with optimal budgeting on TotalSegmentator. (III) We enhanced AI training efficiency and performance by focusing on high-quality pseudo labels, resulting in a 33%--88% performance boost over entropy-based methods, with a cost of 31% time and 4.5% memory. The data and model are released.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 30 of its 52 references that are in the map. First posted on arXiv 1 Jun 2024.
Builds on
30 of 52 references.
References · 30
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
- 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
- 2018Attention U-Net: Learning Where to Look for the Pancreascited by 1,420
- 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 954
- 2017Deep Bayesian Active Learning with Image Datacited by 812
- 2022TotalSegmentator: robust segmentation of 104 anatomical structures in CT imagescited by 597
- 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
18 more
- 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 504
- 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 418
- 2019Learning Loss for Active Learningcited by 335
- 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
- 2020AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?cited by 222
- 2015DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentationcited by 158
- 2019A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysiscited by 150
- 2023STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-trainingcited by 96
- 2018Efficient Active Learning for Image Classification and Segmentation using a Sample Selection and Conditional Generative Adversarial Networkcited by 74
- 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Studycited by 66
- 2023AbdomenAtlas-8K: Annotating 8,000 CT Volumes for Multi-Organ Segmentation in Three Weekscited by 52
- 2023Unleashing the Strengths of Unlabeled Data in Pan-cancer Abdominal Organ Quantification: the FLARE22 Challengecited by 50
- 2023MultiTalent: A Multi-Dataset Approach to Medical Image Segmentationcited by 45
- 2018Leveraging Uncertainty Estimates for Predicting Segmentation Qualitycited by 44
- 2018Real-time Prediction of Segmentation Qualitycited by 23
- 2018Automatic CNN-based detection of cardiac MR motion artefacts using k-space data augmentation and curriculum learningcited by 21
- 2023Towards Unifying Anatomy Segmentation: Automated Generation of a Full-body CT Dataset via Knowledge Aggregation and Anatomical Guidelinescited by 17
- 2018Learning-Based Quality Control for Cardiac MR Imagescited by 4
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.