# Real-Time Panoptic Segmentation from Dense Detections

arXiv: https://arxiv.org/abs/1912.01202 · HTML: https://mlsanity.com/paper/1912.01202/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2019.

## Built on by (most cited first, 30 of 31)

- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Part-aware Panoptic Segmentation](https://mlsanity.com/paper/2106.06351/) (2021)
- [Cross-View Regularization for Domain Adaptive Panoptic Segmentation](https://mlsanity.com/paper/2103.02584/) (2021)
- [You Only Segment Once: Towards Real-Time Panoptic Segmentation](https://mlsanity.com/paper/2303.14651/) (2023)
- [Panoptic-PartFormer: Learning a Unified Model for Panoptic Part Segmentation](https://mlsanity.com/paper/2204.04655/) (2022)
- [PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentation](https://mlsanity.com/paper/2112.02582/) (2021)
- [PEM: Prototype-based Efficient MaskFormer for Image Segmentation](https://mlsanity.com/paper/2402.19422/) (2024)
- [Efficient Multi-Task RGB-D Scene Analysis for Indoor Environments](https://mlsanity.com/paper/2207.04526/) (2022)
- [Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervision](https://mlsanity.com/paper/2108.07682/) (2021)
- [MGNet: Monocular Geometric Scene Understanding for Autonomous Driving](https://mlsanity.com/paper/2206.13199/) (2022)
- [PanopticPartFormer++: A Unified and Decoupled View for Panoptic Part Segmentation](https://mlsanity.com/paper/2301.00954/) (2023)
- [Panoptic Segmentation: A Review](https://mlsanity.com/paper/2111.10250/) (2021)
- [ReMaX: Relaxing for Better Training on Efficient Panoptic Segmentation](https://mlsanity.com/paper/2306.17319/) (2023)
- [PanopticDepth: A Unified Framework for Depth-aware Panoptic Segmentation](https://mlsanity.com/paper/2206.00468/) (2022)
- [SFNet: Faster and Accurate Semantic Segmentation via Semantic Flow](https://mlsanity.com/paper/2207.04415/) (2022)
- [Segmenting Known Objects and Unseen Unknowns without Prior Knowledge](https://mlsanity.com/paper/2209.05407/) (2022)
- [Uncertainty Aware Proposal Segmentation for Unknown Object Detection](https://mlsanity.com/paper/2111.12866/) (2021)
- [Hierarchical Lov'asz Embeddings for Proposal-free Panoptic Segmentation](https://mlsanity.com/paper/2106.04555/) (2021)
- [Camera-Radar Perception for Autonomous Vehicles and ADAS: Concepts, Datasets and Metrics](https://mlsanity.com/paper/2303.04302/) (2023)
- [MaskConver: Revisiting Pure Convolution Model for Panoptic Segmentation](https://mlsanity.com/paper/2312.06052/) (2023)
- [Panoptic SwiftNet: Pyramidal Fusion for Real-time Panoptic Segmentation](https://mlsanity.com/paper/2203.07908/) (2022)
- [Time-Space Transformers for Video Panoptic Segmentation](https://mlsanity.com/paper/2210.03546/) (2022)
- [PanDepth: Joint Panoptic Segmentation and Depth Completion](https://mlsanity.com/paper/2212.14180/) (2022)
- [The revenge of BiSeNet: Efficient Multi-Task Image Segmentation](https://mlsanity.com/paper/2404.09570/) (2024)
- [RT-K-Net: Revisiting K-Net for Real-Time Panoptic Segmentation](https://mlsanity.com/paper/2305.01255/) (2023)
- [Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation](https://mlsanity.com/paper/2404.14657/) (2024)
- [CFNet: Learning Correlation Functions for One-Stage Panoptic Segmentation](https://mlsanity.com/paper/2201.04796/) (2022)
- [Online panoptic 3D reconstruction as a Linear Assignment Problem](https://mlsanity.com/paper/2204.00231/) (2022)
- [SUNet: Scale-aware Unified Network for Panoptic Segmentation](https://mlsanity.com/paper/2209.02877/) (2022)
- [Image-Specific Adaptation of Transformer Encoders for Compute-Efficient Segmentation](https://mlsanity.com/paper/2404.15244/) (2024)

## Abstract (arXiv)

Panoptic segmentation is a complex full scene parsing task requiring simultaneous instance and semantic segmentation at high resolution. Current state-of-the-art approaches cannot run in real-time, and simplifying these architectures to improve efficiency severely degrades their accuracy. In this paper, we propose a new single-shot panoptic segmentation network that leverages dense detections and a global self-attention mechanism to operate in real-time with performance approaching the state of the art. We introduce a novel parameter-free mask construction method that substantially reduces computational complexity by efficiently reusing information from the object detection and semantic segmentation sub-tasks. The resulting network has a simple data flow that does not require feature map re-sampling or clustering post-processing, enabling significant hardware acceleration. Our experiments on the Cityscapes and COCO benchmarks show that our network works at 30 FPS on 1024x2048 resolution, trading a 3% relative performance degradation from the current state of the art for up to 440% faster inference.