# Exemplar-Based Open-Set Panoptic Segmentation Network

arXiv: https://arxiv.org/abs/2105.08336 · HTML: https://mlsanity.com/paper/2105.08336/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 58 references that are in the map.
- First posted on arXiv 18 May 2021.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)

## Built on by (most cited first, 30 of 38)

- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [RbA: Segmenting Unknown Regions Rejected by All](https://mlsanity.com/paper/2211.14293/) (2022)
- [A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future](https://mlsanity.com/paper/2307.09220/) (2023)
- [On Advantages of Mask-level Recognition for Outlier-aware Segmentation](https://mlsanity.com/paper/2301.03407/) (2023)
- [Expanding Low-Density Latent Regions for Open-Set Object Detection](https://mlsanity.com/paper/2203.14911/) (2022)
- [Picture that Sketch: Photorealistic Image Generation from Abstract Sketches](https://mlsanity.com/paper/2303.11162/) (2023)
- [Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation](https://mlsanity.com/paper/2301.00805/) (2023)
- [OpenTAL: Towards Open Set Temporal Action Localization](https://mlsanity.com/paper/2203.05114/) (2022)
- [Panoptic Segmentation: A Review](https://mlsanity.com/paper/2111.10250/) (2021)
- [Mask2Anomaly: Mask Transformer for Universal Open-set Segmentation](https://mlsanity.com/paper/2309.04573/) (2023)
- [SegPrompt: Boosting Open-world Segmentation via Category-level Prompt Learning](https://mlsanity.com/paper/2308.06531/) (2023)
- [UGainS: Uncertainty Guided Anomaly Instance Segmentation](https://mlsanity.com/paper/2308.02046/) (2023)
- [CoGS: Controllable Generation and Search from Sketch and Style](https://mlsanity.com/paper/2203.09554/) (2022)
- [Panoptic Out-of-Distribution Segmentation](https://mlsanity.com/paper/2310.11797/) (2023)
- [Open-Set Recognition in the Age of Vision-Language Models](https://mlsanity.com/paper/2403.16528/) (2024)
- [Learning to Discover and Detect Objects](https://mlsanity.com/paper/2210.10774/) (2022)
- [Spotting the Unexpected (STU): A 3D LiDAR Dataset for Anomaly Segmentation in Autonomous Driving](https://mlsanity.com/paper/2505.02148/) (2025)
- [Dual Decision Improves Open-Set Panoptic Segmentation](https://mlsanity.com/paper/2207.02504/) (2022)
- [Segmenting Known Objects and Unseen Unknowns without Prior Knowledge](https://mlsanity.com/paper/2209.05407/) (2022)
- [OoDIS: Anomaly Instance Segmentation and Detection Benchmark](https://mlsanity.com/paper/2406.11835/) (2024)
- [Lidar Panoptic Segmentation in an Open World](https://mlsanity.com/paper/2409.14273/) (2024)
- [GOSS: Towards Generalized Open-set Semantic Segmentation](https://mlsanity.com/paper/2203.12116/) (2022)
- [Prior2Former -- Evidential Modeling of Mask Transformers for Assumption-Free Open-World Panoptic Segmentation](https://mlsanity.com/paper/2504.04841/) (2025)
- [Open-Set LiDAR Panoptic Segmentation Guided by Uncertainty-Aware Learning](https://mlsanity.com/paper/2506.13265/) (2025)
- [Not All Inputs Are Valid: Towards Open-Set Video Moment Retrieval Using Language](https://mlsanity.com/paper/2605.29812/) (2026)
- [PDF: A Probability-Driven Framework for Open World 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2404.00979/) (2024)
- [Reconstruction guided Meta-learning for Few Shot Open Set Recognition](https://mlsanity.com/paper/2108.00340/) (2021)
- [ElC-OIS: Ellipsoidal Clustering for Open-World Instance Segmentation on LiDAR Data](https://mlsanity.com/paper/2303.04351/) (2023)
- [Open-World Panoptic Segmentation](https://mlsanity.com/paper/2412.12740/) (2024)
- [SketchYourSeg: Mask-Free Subjective Image Segmentation via Freehand Sketches](https://mlsanity.com/paper/2501.16022/) (2025)

## Abstract (arXiv)

We extend panoptic segmentation to the open-world and introduce an open-set panoptic segmentation (OPS) task. This task requires performing panoptic segmentation for not only known classes but also unknown ones that have not been acknowledged during training. We investigate the practical challenges of the task and construct a benchmark on top of an existing dataset, COCO. In addition, we propose a novel exemplar-based open-set panoptic segmentation network (EOPSN) inspired by exemplar theory. Our approach identifies a new class based on exemplars, which are identified by clustering and employed as pseudo-ground-truths. The size of each class increases by mining new exemplars based on the similarities to the existing ones associated with the class. We evaluate EOPSN on the proposed benchmark and demonstrate the effectiveness of our proposals. The primary goal of our work is to draw the attention of the community to the recognition in the open-world scenarios. The implementation of our algorithm is available on the project webpage: https://cv.snu.ac.kr/research/EOPSN.