# Panoptic Out-of-Distribution Segmentation

arXiv: https://arxiv.org/abs/2310.11797 · HTML: https://mlsanity.com/paper/2310.11797/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 18 Oct 2023.

## Builds on (most cited first, 27 of 27)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [Entropy Maximization and Meta Classification for Out-Of-Distribution Detection in Semantic Segmentation](https://mlsanity.com/paper/2012.06575/) (2020)
- [EfficientPS: Efficient Panoptic Segmentation](https://mlsanity.com/paper/2004.02307/) (2020)
- [FreeSOLO: Learning to Segment Objects without Annotations](https://mlsanity.com/paper/2202.12181/) (2022)
- [Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation](https://mlsanity.com/paper/2003.08440/) (2020)
- [RbA: Segmenting Unknown Regions Rejected by All](https://mlsanity.com/paper/2211.14293/) (2022)
- [Residual Pattern Learning for Pixel-wise Out-of-Distribution Detection in Semantic Segmentation](https://mlsanity.com/paper/2211.14512/) (2022)
- [RegNet: Self-Regulated Network for Image Classification](https://mlsanity.com/paper/2101.00590/) (2021)
- [Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation](https://mlsanity.com/paper/2108.01634/) (2021)
- [Exemplar-Based Open-Set Panoptic Segmentation Network](https://mlsanity.com/paper/2105.08336/) (2021)
- [Amodal Panoptic Segmentation](https://mlsanity.com/paper/2202.11542/) (2022)
- [SkyEye: Self-Supervised Bird's-Eye-View Semantic Mapping Using Monocular Frontal View Images](https://mlsanity.com/paper/2302.04233/) (2023)
- [Perceiving the Invisible: Proposal-Free Amodal Panoptic Segmentation](https://mlsanity.com/paper/2205.14637/) (2022)
- [Continual SLAM: Beyond Lifelong Simultaneous Localization and Mapping Through Continual Learning](https://mlsanity.com/paper/2203.01578/) (2022)
- [Dual Decision Improves Open-Set Panoptic Segmentation](https://mlsanity.com/paper/2207.02504/) (2022)
- [Anomaly-Aware Semantic Segmentation via Style-Aligned OoD Augmentation](https://mlsanity.com/paper/2308.09965/) (2023)

## Built on by (most cited first, 14 of 14)

- [AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Driving](https://mlsanity.com/paper/2405.07865/) (2024)
- [Progressive Multi-Modal Fusion for Robust 3D Object Detection](https://mlsanity.com/paper/2410.07475/) (2024)
- [Taxonomy-Aware Continual Semantic Segmentation in Hyperbolic Spaces for Open-World Perception](https://mlsanity.com/paper/2407.18145/) (2024)
- [Hyperspectral Adapter for Semantic Segmentation with Vision Foundation Models](https://mlsanity.com/paper/2509.20107/) (2025)
- [ForecastOcc: Vision-based Semantic Occupancy Forecasting](https://mlsanity.com/paper/2602.08006/) (2026)
- [Open-Set LiDAR Panoptic Segmentation Guided by Uncertainty-Aware Learning](https://mlsanity.com/paper/2506.13265/) (2025)
- [UP-Fuse: Uncertainty-guided LiDAR-Camera Fusion for 3D Panoptic Segmentation](https://mlsanity.com/paper/2602.19349/) (2026)
- [Stochasticity in Motion: An Information-Theoretic Approach to Trajectory Prediction](https://mlsanity.com/paper/2410.01628/) (2024)
- [Evidential Uncertainty Estimation for Multi-Modal Trajectory Prediction](https://mlsanity.com/paper/2503.05274/) (2025)
- [Panoptic-Depth Forecasting](https://mlsanity.com/paper/2409.12008/) (2024)
- [Hyp2Former: Hierarchy-Aware Hyperbolic Embeddings for Open-Set Panoptic Segmentation](https://mlsanity.com/paper/2605.02580/) (2026)
- [U-SEG: Uncertainty in SEGmentation -- A systematic multi-variable exploration](https://mlsanity.com/paper/2605.15421/) (2026)
- [Leveraging Vision-Based Point Cloud Map Priors for Camera-Based 3D Object Detection and Online Vectorized HD Mapping](https://mlsanity.com/paper/2609.26325/) (2026)
- [Perception Beyond the Visible: Incompleteness-Aware Scene Understanding for Robot Autonomy](https://mlsanity.com/paper/2610.18670/) (2026)

## Abstract (arXiv)

Deep learning has led to remarkable strides in scene understanding with panoptic segmentation emerging as a key holistic scene interpretation task. However, the performance of panoptic segmentation is severely impacted in the presence of out-of-distribution (OOD) objects i.e. categories of objects that deviate from the training distribution. To overcome this limitation, we propose Panoptic Out-of Distribution Segmentation for joint pixel-level semantic in-distribution and out-of-distribution classification with instance prediction. We extend two established panoptic segmentation benchmarks, Cityscapes and BDD100K, with out-of-distribution instance segmentation annotations, propose suitable evaluation metrics, and present multiple strong baselines. Importantly, we propose the novel PoDS architecture with a shared backbone, an OOD contextual module for learning global and local OOD object cues, and dual symmetrical decoders with task-specific heads that employ our alignment-mismatch strategy for better OOD generalization. Combined with our data augmentation strategy, this approach facilitates progressive learning of out-of-distribution objects while maintaining in-distribution performance. We perform extensive evaluations that demonstrate that our proposed PoDS network effectively addresses the main challenges and substantially outperforms the baselines. We make the dataset, code, and trained models publicly available at http://pods.cs.uni-freiburg.de.