# SOGNet: Scene Overlap Graph Network for Panoptic Segmentation

arXiv: https://arxiv.org/abs/1911.07527 · HTML: https://mlsanity.com/paper/1911.07527/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2019.

## Built on by (most cited first, 26 of 26)

- [Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentation](https://mlsanity.com/paper/2204.04656/) (2022)
- [CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation](https://mlsanity.com/paper/2206.08948/) (2022)
- [Part-aware Panoptic Segmentation](https://mlsanity.com/paper/2106.06351/) (2021)
- [RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model](https://mlsanity.com/paper/2309.00810/) (2023)
- [Panoptic-PartFormer: Learning a Unified Model for Panoptic Part Segmentation](https://mlsanity.com/paper/2204.04655/) (2022)
- [PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentation](https://mlsanity.com/paper/2112.02582/) (2021)
- [Multi-Task Learning with Multi-Query Transformer for Dense Prediction](https://mlsanity.com/paper/2205.14354/) (2022)
- [Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervision](https://mlsanity.com/paper/2108.07682/) (2021)
- [DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction](https://mlsanity.com/paper/2301.03461/) (2023)
- [PanopticPartFormer++: A Unified and Decoupled View for Panoptic Part Segmentation](https://mlsanity.com/paper/2301.00954/) (2023)
- [Panoptic Segmentation: A Review](https://mlsanity.com/paper/2111.10250/) (2021)
- [Backbones-Review: Feature Extraction Networks for Deep Learning and Deep Reinforcement Learning Approaches](https://mlsanity.com/paper/2206.08016/) (2022)
- [kMaX-DeepLab: k-means Mask Transformer](https://mlsanity.com/paper/2207.04044/) (2022)
- [LiDAR-based 4D Panoptic Segmentation via Dynamic Shifting Network](https://mlsanity.com/paper/2203.07186/) (2022)
- [Combinatorial Optimization for Panoptic Segmentation: A Fully Differentiable Approach](https://mlsanity.com/paper/2106.03188/) (2021)
- [SFNet: Faster and Accurate Semantic Segmentation via Semantic Flow](https://mlsanity.com/paper/2207.04415/) (2022)
- [Panoptic, Instance and Semantic Relations: A Relational Context Encoder to Enhance Panoptic Segmentation](https://mlsanity.com/paper/2204.05370/) (2022)
- [AGNN: Alternating Graph-Regularized Neural Networks to Alleviate Over-Smoothing](https://mlsanity.com/paper/2304.07014/) (2023)
- [Hierarchical Lov'asz Embeddings for Proposal-free Panoptic Segmentation](https://mlsanity.com/paper/2106.04555/) (2021)
- [Intra-Batch Supervision for Panoptic Segmentation on High-Resolution Images](https://mlsanity.com/paper/2304.08222/) (2023)
- [SE-PSNet: Silhouette-based Enhancement Feature for Panoptic Segmentation Network](https://mlsanity.com/paper/2107.05093/) (2021)
- [A Dynamic Feature Interaction Framework for Multi-task Visual Perception](https://mlsanity.com/paper/2306.05061/) (2023)
- [Uncertainty estimation in Deep Learning for Panoptic segmentation](https://mlsanity.com/paper/2304.02098/) (2023)
- [CFNet: Learning Correlation Functions for One-Stage Panoptic Segmentation](https://mlsanity.com/paper/2201.04796/) (2022)
- [SegForestNet: Spatial-Partitioning-Based Aerial Image Segmentation](https://mlsanity.com/paper/2302.01585/) (2023)
- [A Survey on Cell Nuclei Instance Segmentation and Classification: Leveraging Context and Attention](https://mlsanity.com/paper/2407.18673/) (2024)

## Abstract (arXiv)

The panoptic segmentation task requires a unified result from semantic and instance segmentation outputs that may contain overlaps. However, current studies widely ignore modeling overlaps. In this study, we aim to model overlap relations among instances and resolve them for panoptic segmentation. Inspired by scene graph representation, we formulate the overlapping problem as a simplified case, named scene overlap graph. We leverage each object's category, geometry and appearance features to perform relational embedding, and output a relation matrix that encodes overlap relations. In order to overcome the lack of supervision, we introduce a differentiable module to resolve the overlap between any pair of instances. The mask logits after removing overlaps are fed into per-pixel instance verb|id| classification, which leverages the panoptic supervision to assist in the modeling of overlap relations. Besides, we generate an approximate ground truth of overlap relations as the weak supervision, to quantify the accuracy of overlap relations predicted by our method. Experiments on COCO and Cityscapes demonstrate that our method is able to accurately predict overlap relations, and outperform the state-of-the-art performance for panoptic segmentation. Our method also won the Innovation Award in COCO 2019 challenge.