# Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving

arXiv: https://arxiv.org/abs/2105.13688 · HTML: https://mlsanity.com/paper/2105.13688/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 28 May 2021.

## Builds on (most cited first, 20 of 20)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding](https://mlsanity.com/paper/1511.02680/) (2015)
- [Ensemble Distribution Distillation](https://mlsanity.com/paper/1905.00076/) (2019)
- [The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation](https://mlsanity.com/paper/1904.03215/) (2019)
- [Noisy Natural Gradient as Variational Inference](https://mlsanity.com/paper/1712.02390/) (2017)
- [Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation](https://mlsanity.com/paper/1908.00598/) (2019)
- [Bayesian Uncertainty Estimation for Batch Normalized Deep Networks](https://mlsanity.com/paper/1802.06455/) (2018)
- [Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control](https://mlsanity.com/paper/1811.06817/) (2018)
- [Uncertainty-Aware Learning from Demonstration using Mixture Density Networks with Sampling-Free Variance Modeling](https://mlsanity.com/paper/1709.02249/) (2017)
- [Inhibited Softmax for Uncertainty Estimation in Neural Networks](https://mlsanity.com/paper/1810.01861/) (2018)
- [Dropout Distillation for Efficiently Estimating Model Confidence](https://mlsanity.com/paper/1809.10562/) (2018)

## Built on by (most cited first, 8 of 8)

- [Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation](https://mlsanity.com/paper/2108.01634/) (2021)
- [DUDES: Deep Uncertainty Distillation using Ensembles for Semantic Segmentation](https://mlsanity.com/paper/2303.09843/) (2023)
- [ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models](https://mlsanity.com/paper/2307.00398/) (2023)
- [Uncertainty Estimation for Safety-critical Scene Segmentation via Fine-grained Reward Maximization](https://mlsanity.com/paper/2311.02719/) (2023)
- [Efficient Multi-task Uncertainties for Joint Semantic Segmentation and Monocular Depth Estimation](https://mlsanity.com/paper/2402.10580/) (2024)
- [WeedsGalore: A Multispectral and Multitemporal UAV-based Dataset for Crop and Weed Segmentation in Agricultural Maize Fields](https://mlsanity.com/paper/2502.13103/) (2025)
- [Trust It or Not: Evidential Uncertainty for Feed-Forward 3D Reconstruction with Trust3R](https://mlsanity.com/paper/2605.19539/) (2026)
- [Instance-Aware Observer Network for Out-of-Distribution Object Segmentation](https://mlsanity.com/paper/2207.08782/) (2022)

## Abstract (arXiv)

In this paper, we show how uncertainty estimation can be leveraged to enable safety critical image segmentation in autonomous driving, by triggering a fallback behavior if a target accuracy cannot be guaranteed. We introduce a new uncertainty measure based on disagreeing predictions as measured by a dissimilarity function. We propose to estimate this dissimilarity by training a deep neural architecture in parallel to the task-specific network. It allows this observer to be dedicated to the uncertainty estimation, and let the task-specific network make predictions. We propose to use self-supervision to train the observer, which implies that our method does not require additional training data. We show experimentally that our proposed approach is much less computationally intensive at inference time than competing methods (e.g. MCDropout), while delivering better results on safety-oriented evaluation metrics on the CamVid dataset, especially in the case of glare artifacts.