# Pixel-wise Gradient Uncertainty for Convolutional Neural Networks applied to Out-of-Distribution Segmentation

arXiv: https://arxiv.org/abs/2303.06920 · HTML: https://mlsanity.com/paper/2303.06920/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 56 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 30 of 41)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [On the Importance of Gradients for Detecting Distributional Shifts in the Wild](https://mlsanity.com/paper/2110.00218/) (2021)
- [Wider or Deeper: Revisiting the ResNet Model for Visual Recognition](https://mlsanity.com/paper/1611.10080/) (2016)
- [SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation](https://mlsanity.com/paper/2104.14812/) (2021)
- [Evaluating Bayesian Deep Learning Methods for Semantic Segmentation](https://mlsanity.com/paper/1811.12709/) (2018)
- [Detecting the Unexpected via Image Resynthesis](https://mlsanity.com/paper/1904.07595/) (2019)
- [Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow](https://mlsanity.com/paper/1802.07095/) (2018)
- [Improving Semantic Segmentation via Video Propagation and Label Relaxation](https://mlsanity.com/paper/1812.01593/) (2018)
- [Entropy Maximization and Meta Classification for Out-Of-Distribution Detection in Semantic Segmentation](https://mlsanity.com/paper/2012.06575/) (2020)
- [The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation](https://mlsanity.com/paper/1904.03215/) (2019)
- [Lost and Found: Detecting Small Road Hazards for Self-Driving Vehicles](https://mlsanity.com/paper/1609.04653/) (2016)
- [Pixel-wise Anomaly Detection in Complex Driving Scenes](https://mlsanity.com/paper/2103.05445/) (2021)
- [Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes](https://mlsanity.com/paper/2111.12264/) (2021)
- [RbA: Segmenting Unknown Regions Rejected by All](https://mlsanity.com/paper/2211.14293/) (2022)
- [Residual Pattern Learning for Pixel-wise Out-of-Distribution Detection in Semantic Segmentation](https://mlsanity.com/paper/2211.14512/) (2022)
- [Prediction Error Meta Classification in Semantic Segmentation: Detection via Aggregated Dispersion Measures of Softmax Probabilities](https://mlsanity.com/paper/1811.00648/) (2018)
- [Leveraging Uncertainty Estimates for Predicting Segmentation Quality](https://mlsanity.com/paper/1807.00502/) (2018)
- [Classification Uncertainty of Deep Neural Networks Based on Gradient Information](https://mlsanity.com/paper/1805.08440/) (2018)
- [DenseHybrid: Hybrid Anomaly Detection for Dense Open-set Recognition](https://mlsanity.com/paper/2207.02606/) (2022)
- [On Advantages of Mask-level Recognition for Outlier-aware Segmentation](https://mlsanity.com/paper/2301.03407/) (2023)

## Built on by (most cited first, 11 of 11)

- [Open-World Semantic Segmentation Including Class Similarity](https://mlsanity.com/paper/2403.07532/) (2024)
- [Out-of-Distribution Segmentation in Autonomous Driving: Problems and State of the Art](https://mlsanity.com/paper/2503.08695/) (2025)
- [Uncertainty-weighted Loss Functions for Improved Adversarial Attacks on Semantic Segmentation](https://mlsanity.com/paper/2310.17436/) (2023)
- [Revisiting Gradient-based Uncertainty for Monocular Depth Estimation](https://mlsanity.com/paper/2502.05964/) (2025)
- [Multi-Scale Foreground-Background Confidence for Out-of-Distribution Segmentation](https://mlsanity.com/paper/2412.16990/) (2024)
- [Uncertainty estimates for semantic segmentation: providing enhanced reliability for automated motor claims handling](https://mlsanity.com/paper/2401.09245/) (2024)
- [Open-World Panoptic Segmentation](https://mlsanity.com/paper/2412.12740/) (2024)
- [Uncertainty evaluation of segmentation models for Earth observation](https://mlsanity.com/paper/2510.19586/) (2025)
- [Out-of-Distribution Segmentation via Wasserstein-Based Evidential Uncertainty](https://mlsanity.com/paper/2512.11373/) (2025)
- [Towards Reliable Detection of Empty Space: Conditional Marked Point Processes for Object Detection](https://mlsanity.com/paper/2506.21486/) (2025)
- [Combining Foundation Model Confidence and Monocular Depth for Training-Free Out-of-Distribution Segmentation](https://mlsanity.com/paper/2609.22896/) (2026)

## Abstract (arXiv)

In recent years, deep neural networks have defined the state-of-the-art in semantic segmentation where their predictions are constrained to a predefined set of semantic classes. They are to be deployed in applications such as automated driving, although their categorically confined expressive power runs contrary to such open world scenarios. Thus, the detection and segmentation of objects from outside their predefined semantic space, i.e., out-of-distribution (OoD) objects, is of highest interest. Since uncertainty estimation methods like softmax entropy or Bayesian models are sensitive to erroneous predictions, these methods are a natural baseline for OoD detection. Here, we present a method for obtaining uncertainty scores from pixel-wise loss gradients which can be computed efficiently during inference. Our approach is simple to implement for a large class of models, does not require any additional training or auxiliary data and can be readily used on pre-trained segmentation models. Our experiments show the ability of our method to identify wrong pixel classifications and to estimate prediction quality at negligible computational overhead. In particular, we observe superior performance in terms of OoD segmentation to comparable baselines on the SegmentMeIfYouCan benchmark, clearly outperforming other methods.