# Uncertainty-aware Panoptic Segmentation

arXiv: https://arxiv.org/abs/2206.14554 · HTML: https://mlsanity.com/paper/2206.14554/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 41 references that are in the map.
- First posted on arXiv 29 Jun 2022.

## Builds on (most cited first, 30 of 35)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Deep Learning for Generic Object Detection: A Survey](https://mlsanity.com/paper/1809.02165/) (2018)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation](https://mlsanity.com/paper/2003.14032/) (2020)
- [UPSNet: A Unified Panoptic Segmentation Network](https://mlsanity.com/paper/1901.03784/) (2019)
- [IDD: A Dataset for Exploring Problems of Autonomous Navigation in Unconstrained Environments](https://mlsanity.com/paper/1811.10200/) (2018)
- [Seamless Scene Segmentation](https://mlsanity.com/paper/1905.01220/) (2019)
- [EfficientPS: Efficient Panoptic Segmentation](https://mlsanity.com/paper/2004.02307/) (2020)
- [Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentation](https://mlsanity.com/paper/2103.14962/) (2021)
- [Searching for Efficient Multi-Scale Architectures for Dense Image Prediction](https://mlsanity.com/paper/1809.04184/) (2018)
- [SSAP: Single-Shot Instance Segmentation With Affinity Pyramid](https://mlsanity.com/paper/1909.01616/) (2019)
- [EfficientLPS: Efficient LiDAR Panoptic Segmentation](https://mlsanity.com/paper/2102.08009/) (2021)
- [Attention-guided Unified Network for Panoptic Segmentation](https://mlsanity.com/paper/1812.03904/) (2018)
- [Panoptic-PHNet: Towards Real-Time and High-Precision LiDAR Panoptic Segmentation via Clustering Pseudo Heatmap](https://mlsanity.com/paper/2205.07002/) (2022)
- [Part-aware Panoptic Segmentation](https://mlsanity.com/paper/2106.06351/) (2021)
- [Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images](https://mlsanity.com/paper/2108.03227/) (2021)
- [Efficient Uncertainty Estimation for Semantic Segmentation in Videos](https://mlsanity.com/paper/1807.11037/) (2018)
- [HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Images](https://mlsanity.com/paper/2003.04645/) (2020)
- [A Benchmark for LiDAR-based Panoptic Segmentation based on KITTI](https://mlsanity.com/paper/2003.02371/) (2020)

## Built on by (most cited first, 17 of 17)

- [MUSES: The Multi-Sensor Semantic Perception Dataset for Driving under Uncertainty](https://mlsanity.com/paper/2401.12761/) (2024)
- [A Comprehensive Survey on Evidential Deep Learning and Its Applications](https://mlsanity.com/paper/2409.04720/) (2024)
- [WildScenes: A Benchmark for 2D and 3D Semantic Segmentation in Large-scale Natural Environments](https://mlsanity.com/paper/2312.15364/) (2023)
- [EvCenterNet: Uncertainty Estimation for Object Detection using Evidential Learning](https://mlsanity.com/paper/2303.03037/) (2023)
- [Segmenting Known Objects and Unseen Unknowns without Prior Knowledge](https://mlsanity.com/paper/2209.05407/) (2022)
- [Uncertainty-aware LiDAR Panoptic Segmentation](https://mlsanity.com/paper/2210.04472/) (2022)
- [Evidential Semantic Mapping in Off-road Environments with Uncertainty-aware Bayesian Kernel Inference](https://mlsanity.com/paper/2403.14138/) (2024)
- [Perception Matters: Enhancing Embodied AI with Uncertainty-Aware Semantic Segmentation](https://mlsanity.com/paper/2408.02297/) (2024)
- [Evidential Uncertainty Estimation for Multi-Modal Trajectory Prediction](https://mlsanity.com/paper/2503.05274/) (2025)
- [ProPanDL: A Modular Architecture for Uncertainty-Aware Panoptic Segmentation](https://mlsanity.com/paper/2304.08645/) (2023)
- [Uncertainty-aware Semantic Mapping in Off-road Environments with Dempster-Shafer Theory of Evidence](https://mlsanity.com/paper/2405.06265/) (2024)
- [Anomalies by Synthesis: Anomaly Detection using Generative Diffusion Models for Off-Road Navigation](https://mlsanity.com/paper/2505.22805/) (2025)
- [NRSeg: Noise-Resilient Learning for BEV Semantic Segmentation via Driving World Models](https://mlsanity.com/paper/2507.04002/) (2025)
- [Uncertainty estimation in Deep Learning for Panoptic segmentation](https://mlsanity.com/paper/2304.02098/) (2023)
- [A Smart Robotic System for Industrial Plant Supervision](https://mlsanity.com/paper/2308.05612/) (2023)
- [U-SEG: Uncertainty in SEGmentation -- A systematic multi-variable exploration](https://mlsanity.com/paper/2605.15421/) (2026)
- [HGeo-TopoMap: Boosting Topological Mapping with Hierarchical Geometric Priors](https://mlsanity.com/paper/2607.21281/) (2026)

## Abstract (arXiv)

Reliable scene understanding is indispensable for modern autonomous systems. Current learning-based methods typically try to maximize their performance based on segmentation metrics that only consider the quality of the segmentation. However, for the safe operation of a system in the real world it is crucial to consider the uncertainty in the prediction as well. In this work, we introduce the novel task of uncertainty-aware panoptic segmentation, which aims to predict per-pixel semantic and instance segmentations, together with per-pixel uncertainty estimates. We define two novel metrics to facilitate its quantitative analysis, the uncertainty-aware Panoptic Quality (uPQ) and the panoptic Expected Calibration Error (pECE). We further propose the novel top-down Evidential Panoptic Segmentation Network (EvPSNet) to solve this task. Our architecture employs a simple yet effective panoptic fusion module that leverages the predicted uncertainties. Furthermore, we provide several strong baselines combining state-of-the-art panoptic segmentation networks with sampling-free uncertainty estimation techniques. Extensive evaluations show that our EvPSNet achieves the new state-of-the-art for the standard Panoptic Quality (PQ), as well as for our uncertainty-aware panoptic metrics. We make the code available at: urlhttps://github.com/kshitij3112/EvPSNet