# Panoptic Segmentation: A Review

arXiv: https://arxiv.org/abs/2111.10250 · HTML: https://mlsanity.com/paper/2111.10250/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 85 of its 177 references that are in the map.
- First posted on arXiv 19 Nov 2021.

## Builds on (most cited first, 30 of 85)

- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [Domain Adaptive Faster R-CNN for Object Detection in the Wild](https://mlsanity.com/paper/1803.03243/) (2018)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution](https://mlsanity.com/paper/2006.02334/) (2020)
- [UPSNet: A Unified Panoptic Segmentation Network](https://mlsanity.com/paper/1901.03784/) (2019)
- [Understanding Convolution for Semantic Segmentation](https://mlsanity.com/paper/1702.08502/) (2017)
- [Video Panoptic Segmentation](https://mlsanity.com/paper/2006.11339/) (2020)
- [InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring](https://mlsanity.com/paper/2103.01128/) (2021)
- [Diversify and Match: A Domain Adaptive Representation Learning Paradigm for Object Detection](https://mlsanity.com/paper/1905.05396/) (2019)
- [Synergistic Image and Feature Adaptation: Towards Cross-Modality Domain Adaptation for Medical Image Segmentation](https://mlsanity.com/paper/1901.08211/) (2019)
- [Seamless Scene Segmentation](https://mlsanity.com/paper/1905.01220/) (2019)
- [EfficientPS: Efficient Panoptic Segmentation](https://mlsanity.com/paper/2004.02307/) (2020)
- [Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentation](https://mlsanity.com/paper/2103.14962/) (2021)
- [PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things](https://mlsanity.com/paper/1903.01177/) (2019)
- [Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth](https://mlsanity.com/paper/1906.11109/) (2019)

## Built on by (most cited first, 18 of 18)

- [CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenes](https://mlsanity.com/paper/2410.10791/) (2024)
- [OmniSAM: Omnidirectional Segment Anything Model for UDA in Panoramic Semantic Segmentation](https://mlsanity.com/paper/2503.07098/) (2025)
- [FoodSAM: Any Food Segmentation](https://mlsanity.com/paper/2308.05938/) (2023)
- [AI in Thyroid Cancer Diagnosis: Techniques, Trends, and Future Directions](https://mlsanity.com/paper/2308.13592/) (2023)
- [Backbones-Review: Feature Extraction Networks for Deep Learning and Deep Reinforcement Learning Approaches](https://mlsanity.com/paper/2206.08016/) (2022)
- [MMSFormer: Multimodal Transformer for Material and Semantic Segmentation](https://mlsanity.com/paper/2309.04001/) (2023)
- [Reasoning Segmentation for Images and Videos: A Survey](https://mlsanity.com/paper/2505.18816/) (2025)
- [Improving CNN-based Person Re-identification using score Normalization](https://mlsanity.com/paper/2307.00397/) (2023)
- [TRACE: Your Diffusion Model is Secretly an Instance Edge Detector](https://mlsanity.com/paper/2503.07982/) (2025)
- [Deep Transfer Learning for Kidney Cancer Diagnosis](https://mlsanity.com/paper/2408.04318/) (2024)
- [3D objects and scenes classification, recognition, segmentation, and reconstruction using 3D point cloud data: A review](https://mlsanity.com/paper/2306.05978/) (2023)
- [Machine Learning and Transformers for Thyroid Carcinoma Diagnosis: A Review](https://mlsanity.com/paper/2403.13843/) (2024)
- [Agricultural Landscape Understanding At Country-Scale](https://mlsanity.com/paper/2411.05359/) (2024)
- [MaSS13K: A Matting-level Semantic Segmentation Benchmark](https://mlsanity.com/paper/2503.18364/) (2025)
- [SO3UFormer: Learning Intrinsic Spherical Features for Rotation-Robust Panoramic Dense Prediction](https://mlsanity.com/paper/2602.22867/) (2026)
- [Scene-Centric Unsupervised Video Panoptic Segmentation](https://mlsanity.com/paper/2606.04925/) (2026)
- [StableSemantics: A Synthetic Language-Vision Dataset of Semantic Representations in Naturalistic Images](https://mlsanity.com/paper/2406.13735/) (2024)
- [A Review of YOLOv12: Attention-Based Enhancements vs. Previous Versions](https://mlsanity.com/paper/2504.11995/) (2025)

## Abstract (arXiv)

Image segmentation for video analysis plays an essential role in different research fields such as smart city, healthcare, computer vision and geoscience, and remote sensing applications. In this regard, a significant effort has been devoted recently to developing novel segmentation strategies; one of the latest outstanding achievements is panoptic segmentation. The latter has resulted from the fusion of semantic and instance segmentation. Explicitly, panoptic segmentation is currently under study to help gain a more nuanced knowledge of the image scenes for video surveillance, crowd counting, self-autonomous driving, medical image analysis, and a deeper understanding of the scenes in general. To that end, we present in this paper the first comprehensive review of existing panoptic segmentation methods to the best of the authors' knowledge. Accordingly, a well-defined taxonomy of existing panoptic techniques is performed based on the nature of the adopted algorithms, application scenarios, and primary objectives. Moreover, the use of panoptic segmentation for annotating new datasets by pseudo-labeling is discussed. Moving on, ablation studies are carried out to understand the panoptic methods from different perspectives. Moreover, evaluation metrics suitable for panoptic segmentation are discussed, and a comparison of the performance of existing solutions is provided to inform the state-of-the-art and identify their limitations and strengths. Lastly, the current challenges the subject technology faces and the future trends attracting considerable interest in the near future are elaborated, which can be a starting point for the upcoming research studies. The papers provided with code are available at: https://github.com/elharroussomar/Awesome-Panoptic-Segmentation