# Automated Detection of Label Errors in Semantic Segmentation Datasets via Deep Learning and Uncertainty Quantification

arXiv: https://arxiv.org/abs/2207.06104 · HTML: https://mlsanity.com/paper/2207.06104/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 49 references that are in the map.
- First posted on arXiv 13 Jul 2022.

## Builds on (most cited first, 30 of 35)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation](https://mlsanity.com/paper/1603.05959/) (2016)
- [Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise](https://mlsanity.com/paper/1802.05300/) (2018)
- [High-Resolution Representations for Labeling Pixels and Regions](https://mlsanity.com/paper/1904.04514/) (2019)
- [Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels](https://mlsanity.com/paper/1905.05040/) (2019)
- [Wider or Deeper: Revisiting the ResNet Model for Visual Recognition](https://mlsanity.com/paper/1611.10080/) (2016)
- [Hierarchical Multi-Scale Attention for Semantic Segmentation](https://mlsanity.com/paper/2005.10821/) (2020)
- [SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation](https://mlsanity.com/paper/2104.14812/) (2021)
- [An overview of deep learning in medical imaging focusing on MRI](https://mlsanity.com/paper/1811.10052/) (2018)
- [Improving Semantic Segmentation via Video Propagation and Label Relaxation](https://mlsanity.com/paper/1812.01593/) (2018)
- [A Survey of Deep Learning Applications to Autonomous Vehicle Control](https://mlsanity.com/paper/1912.10773/) (2019)
- [Disentangling Human Error from the Ground Truth in Segmentation of Medical Images](https://mlsanity.com/paper/2007.15963/) (2020)
- [Prediction Error Meta Classification in Semantic Segmentation: Detection via Aggregated Dispersion Measures of Softmax Probabilities](https://mlsanity.com/paper/1811.00648/) (2018)
- [MetaBox+: A new Region Based Active Learning Method for Semantic Segmentation using Priority Maps](https://mlsanity.com/paper/2010.01884/) (2020)
- [Time-Dynamic Estimates of the Reliability of Deep Semantic Segmentation Networks](https://mlsanity.com/paper/1911.05075/) (2019)
- [Uncertainty Measures and Prediction Quality Rating for the Semantic Segmentation of Nested Multi Resolution Street Scene Images](https://mlsanity.com/paper/1904.04516/) (2019)

## Built on by (most cited first, 20 of 20)

- [Identifying Label Errors in Object Detection Datasets by Loss Inspection](https://mlsanity.com/paper/2303.06999/) (2023)
- [Identifying Incorrect Annotations in Multi-Label Classification Data](https://mlsanity.com/paper/2211.13895/) (2022)
- [Estimating label quality and errors in semantic segmentation data via any model](https://mlsanity.com/paper/2307.05080/) (2023)
- [ObjectLab: Automated Diagnosis of Mislabeled Images in Object Detection Data](https://mlsanity.com/paper/2309.00832/) (2023)
- [Data Cleaning and Machine Learning: A Systematic Literature Review](https://mlsanity.com/paper/2310.01765/) (2023)
- [Active Label Correction for Semantic Segmentation with Foundation Models](https://mlsanity.com/paper/2403.10820/) (2024)
- [Synthetic Data-based Detection of Zebras in Drone Imagery](https://mlsanity.com/paper/2305.00432/) (2023)
- [LEMoN: Label Error Detection using Multimodal Neighbors](https://mlsanity.com/paper/2407.18941/) (2024)
- [Semi-supervised domain adaptation with CycleGAN guided by a downstream task loss](https://mlsanity.com/paper/2208.08815/) (2022)
- [Deep Active Learning with Noisy Oracle in Object Detection](https://mlsanity.com/paper/2310.00372/) (2023)
- [Bridging Annotation Gaps: Transferring Labels to Align Object Detection Datasets](https://mlsanity.com/paper/2506.04737/) (2025)
- [Perception Without Vision for Trajectory Prediction: Ego Vehicle Dynamics as Scene Representation for Efficient Active Learning in Autonomous Driving](https://mlsanity.com/paper/2405.09049/) (2024)
- [A2LC: Active and Automated Label Correction for Semantic Segmentation](https://mlsanity.com/paper/2506.11599/) (2025)
- [Impact of Inaccurate Contamination Ratio on Robust Unsupervised Anomaly Detection](https://mlsanity.com/paper/2408.07718/) (2024)
- [Uncertainty and Prediction Quality Estimation for Semantic Segmentation via Graph Neural Networks](https://mlsanity.com/paper/2409.11373/) (2024)
- [From Label Error Detection to Correction: A Modular Framework and Benchmark for Object Detection Datasets](https://mlsanity.com/paper/2508.06556/) (2025)
- [Learning to Detect Label Errors by Making Them: A Method for Segmentation and Object Detection Datasets](https://mlsanity.com/paper/2508.17930/) (2025)
- [Data-Centric Benchmark for Label Noise Estimation and Ranking in Remote Sensing Binary Building Segmentation](https://mlsanity.com/paper/2603.00604/) (2026)
- [A Multi-Annotator Study of Segmentation Noise and Uncertainty in Turbid Underwater Images](https://mlsanity.com/paper/2608.15363/) (2026)
- [GERIS: A Game-Theoretic Framework for Filtering Instance-Dependent Label Noise in License Plate Data Augmentation](https://mlsanity.com/paper/2609.31731/) (2026)

## Abstract (arXiv)

In this work, we for the first time present a method for detecting label errors in image datasets with semantic segmentation, i.e., pixel-wise class labels. Annotation acquisition for semantic segmentation datasets is time-consuming and requires plenty of human labor. In particular, review processes are time consuming and label errors can easily be overlooked by humans. The consequences are biased benchmarks and in extreme cases also performance degradation of deep neural networks (DNNs) trained on such datasets. DNNs for semantic segmentation yield pixel-wise predictions, which makes detection of label errors via uncertainty quantification a complex task. Uncertainty is particularly pronounced at the transitions between connected components of the prediction. By lifting the consideration of uncertainty to the level of predicted components, we enable the usage of DNNs together with component-level uncertainty quantification for the detection of label errors. We present a principled approach to benchmarking the task of label error detection by dropping labels from the Cityscapes dataset as well from a dataset extracted from the CARLA driving simulator, where in the latter case we have the labels under control. Our experiments show that our approach is able to detect the vast majority of label errors while controlling the number of false label error detections. Furthermore, we apply our method to semantic segmentation datasets frequently used by the computer vision community and present a collection of label errors along with sample statistics.