# PUPS: Point Cloud Unified Panoptic Segmentation

arXiv: https://arxiv.org/abs/2302.06185 · HTML: https://mlsanity.com/paper/2302.06185/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 13 Feb 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [K-Net: Towards Unified Image Segmentation](https://mlsanity.com/paper/2106.14855/) (2021)
- [RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation](https://mlsanity.com/paper/2103.12978/) (2021)
- [Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Tracking](https://mlsanity.com/paper/2109.03805/) (2021)
- [Seamless Scene Segmentation](https://mlsanity.com/paper/1905.01220/) (2019)
- [Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers](https://mlsanity.com/paper/2109.03814/) (2021)
- [Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentation](https://mlsanity.com/paper/2103.14962/) (2021)
- [Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth](https://mlsanity.com/paper/1906.11109/) (2019)
- [LiDAR-based Panoptic Segmentation via Dynamic Shifting Network](https://mlsanity.com/paper/2011.11964/) (2020)
- [EfficientLPS: Efficient LiDAR Panoptic Segmentation](https://mlsanity.com/paper/2102.08009/) (2021)
- [Attention-guided Unified Network for Panoptic Segmentation](https://mlsanity.com/paper/1812.03904/) (2018)
- [Panoptic-PHNet: Towards Real-Time and High-Precision LiDAR Panoptic Segmentation via Clustering Pseudo Heatmap](https://mlsanity.com/paper/2205.07002/) (2022)
- [MOPT: Multi-Object Panoptic Tracking](https://mlsanity.com/paper/2004.08189/) (2020)
- [An End-to-End Network for Panoptic Segmentation](https://mlsanity.com/paper/1903.05027/) (2019)
- [Panoster: End-to-end Panoptic Segmentation of LiDAR Point Clouds](https://mlsanity.com/paper/2010.15157/) (2020)
- [A Benchmark for LiDAR-based Panoptic Segmentation based on KITTI](https://mlsanity.com/paper/2003.02371/) (2020)
- [GP-S3Net: Graph-based Panoptic Sparse Semantic Segmentation Network](https://mlsanity.com/paper/2108.08401/) (2021)

## Built on by (most cited first, 16 of 16)

- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [Mask4Former: Mask Transformer for 4D Panoptic Segmentation](https://mlsanity.com/paper/2309.16133/) (2023)
- [LiDAR-Camera Panoptic Segmentation via Geometry-Consistent and Semantic-Aware Alignment](https://mlsanity.com/paper/2308.01686/) (2023)
- [Position-Guided Point Cloud Panoptic Segmentation Transformer](https://mlsanity.com/paper/2303.13509/) (2023)
- [4D-Former: Multimodal 4D Panoptic Segmentation](https://mlsanity.com/paper/2311.01520/) (2023)
- [How Do Images Align and Complement LiDAR? Towards a Harmonized Multi-modal 3D Panoptic Segmentation](https://mlsanity.com/paper/2505.18956/) (2025)
- [Beyond the Label Itself: Latent Labels Enhance Semi-supervised Point Cloud Panoptic Segmentation](https://mlsanity.com/paper/2312.08234/) (2023)
- [Open-Set LiDAR Panoptic Segmentation Guided by Uncertainty-Aware Learning](https://mlsanity.com/paper/2506.13265/) (2025)
- [UP-Fuse: Uncertainty-guided LiDAR-Camera Fusion for 3D Panoptic Segmentation](https://mlsanity.com/paper/2602.19349/) (2026)
- [Interactive4D: Interactive 4D LiDAR Segmentation](https://mlsanity.com/paper/2410.08206/) (2024)
- [Exploring Scene Affinity for Semi-Supervised LiDAR Semantic Segmentation](https://mlsanity.com/paper/2408.11280/) (2024)
- [Radar Tracker: Moving Instance Tracking in Sparse and Noisy Radar Point Clouds](https://mlsanity.com/paper/2507.03441/) (2025)
- [SemRaFiner: Panoptic Segmentation in Sparse and Noisy Radar Point Clouds](https://mlsanity.com/paper/2507.06906/) (2025)
- [Is clustering enough for LiDAR instance segmentation? A state-of-the-art training-free baseline](https://mlsanity.com/paper/2503.13203/) (2025)
- [Panoptic-CUDAL: Rural Australia Point Cloud Dataset in Rainy Conditions](https://mlsanity.com/paper/2503.16378/) (2025)
- [DQFormer: Towards Unified LiDAR Panoptic Segmentation with Decoupled Queries](https://mlsanity.com/paper/2408.15813/) (2024)

## Abstract (arXiv)

Point cloud panoptic segmentation is a challenging task that seeks a holistic solution for both semantic and instance segmentation to predict groupings of coherent points. Previous approaches treat semantic and instance segmentation as surrogate tasks, and they either use clustering methods or bounding boxes to gather instance groupings with costly computation and hand-crafted designs in the instance segmentation task. In this paper, we propose a simple but effective point cloud unified panoptic segmentation (PUPS) framework, which use a set of point-level classifiers to directly predict semantic and instance groupings in an end-to-end manner. To realize PUPS, we introduce bipartite matching to our training pipeline so that our classifiers are able to exclusively predict groupings of instances, getting rid of hand-crafted designs, e.g. anchors and Non-Maximum Suppression (NMS). In order to achieve better grouping results, we utilize a transformer decoder to iteratively refine the point classifiers and develop a context-aware CutMix augmentation to overcome the class imbalance problem. As a result, PUPS achieves 1st place on the leader board of SemanticKITTI panoptic segmentation task and state-of-the-art results on nuScenes.