# Dual Decision Improves Open-Set Panoptic Segmentation

arXiv: https://arxiv.org/abs/2207.02504 · HTML: https://mlsanity.com/paper/2207.02504/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 6 Jul 2022.

## Builds on (most cited first, 27 of 27)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [Recent Advances in Open Set Recognition: A Survey](https://mlsanity.com/paper/1811.08581/) (2018)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [K-Net: Towards Unified Image Segmentation](https://mlsanity.com/paper/2106.14855/) (2021)
- [UPSNet: A Unified Panoptic Segmentation Network](https://mlsanity.com/paper/1901.03784/) (2019)
- [Classification-Reconstruction Learning for Open-Set Recognition](https://mlsanity.com/paper/1812.04246/) (2018)
- [C2AE: Class Conditioned Auto-Encoder for Open-set Recognition](https://mlsanity.com/paper/1904.01198/) (2019)
- [Entropy Maximization and Meta Classification for Out-Of-Distribution Detection in Semantic Segmentation](https://mlsanity.com/paper/2012.06575/) (2020)
- [OW-DETR: Open-world Detection Transformer](https://mlsanity.com/paper/2112.01513/) (2021)
- [The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation](https://mlsanity.com/paper/1904.03215/) (2019)
- [Fully Convolutional Networks for Panoptic Segmentation](https://mlsanity.com/paper/2012.00720/) (2020)
- [Open-World Entity Segmentation](https://mlsanity.com/paper/2107.14228/) (2021)
- [Attention-guided Unified Network for Panoptic Segmentation](https://mlsanity.com/paper/1812.03904/) (2018)
- [Exemplar-Based Open-Set Panoptic Segmentation Network](https://mlsanity.com/paper/2105.08336/) (2021)
- [Expanding Low-Density Latent Regions for Open-Set Object Detection](https://mlsanity.com/paper/2203.14911/) (2022)
- [Dense outlier detection and open-set recognition based on training with noisy negative images](https://mlsanity.com/paper/2101.09193/) (2021)
- [The surprising impact of mask-head architecture on novel class segmentation](https://mlsanity.com/paper/2104.00613/) (2021)
- [CA-SSL: Class-Agnostic Semi-Supervised Learning for Detection and Segmentation](https://mlsanity.com/paper/2112.04966/) (2021)

## Built on by (most cited first, 9 of 9)

- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [Panoptic Out-of-Distribution Segmentation](https://mlsanity.com/paper/2310.11797/) (2023)
- [Prior2Former -- Evidential Modeling of Mask Transformers for Assumption-Free Open-World Panoptic Segmentation](https://mlsanity.com/paper/2504.04841/) (2025)
- [Towards Universal Vision-language Omni-supervised Segmentation](https://mlsanity.com/paper/2303.06547/) (2023)
- [A Bottom-Up Approach to Class-Agnostic Image Segmentation](https://mlsanity.com/paper/2409.13687/) (2024)
- [A Machine Learning Perspective on Automated Driving Corner Cases](https://mlsanity.com/paper/2510.10653/) (2025)
- [Hyp2Former: Hierarchy-Aware Hyperbolic Embeddings for Open-Set Panoptic Segmentation](https://mlsanity.com/paper/2605.02580/) (2026)
- [FuTCR: Future-Targeted Contrast and Repulsion for Continual Panoptic Segmentation](https://mlsanity.com/paper/2605.12451/) (2026)
- [Perception Beyond the Visible: Incompleteness-Aware Scene Understanding for Robot Autonomy](https://mlsanity.com/paper/2610.18670/) (2026)

## Abstract (arXiv)

Open-set panoptic segmentation (OPS) problem is a new research direction aiming to perform segmentation for both known classes and unknown classes, i.e., the objects ("things") that are never annotated in the training set. The main challenges of OPS are twofold: (1) the infinite possibility of the unknown object appearances makes it difficult to model them from a limited number of training data. (2) at training time, we are only provided with the "void" category, which essentially mixes the "unknown thing" and "background" classes. We empirically find that directly using "void" category to supervise known class or "background" classifiers without screening will lead to an unsatisfied OPS result. In this paper, we propose a divide-and-conquer scheme to develop a dual decision process for OPS. We show that by properly combining a known class discriminator with an additional class-agnostic object prediction head, the OPS performance can be significantly improved. Specifically, we first propose to create a classifier with only known categories and let the "void" class proposals achieve low prediction probability from those categories. Then we distinguish the "unknown things" from the background by using the additional object prediction head. To further boost performance, we introduce "unknown things" pseudo-labels generated from up-to-date models to enrich the training set. Our extensive experimental evaluation shows that our approach significantly improves unknown class panoptic quality, with more than 30% relative improvements than the existing best-performed method.