# Panoptic-PHNet: Towards Real-Time and High-Precision LiDAR Panoptic Segmentation via Clustering Pseudo Heatmap

arXiv: https://arxiv.org/abs/2205.07002 · HTML: https://mlsanity.com/paper/2205.07002/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 42 references that are in the map.
- First posted on arXiv 14 May 2022.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds](https://mlsanity.com/paper/1911.11236/) (2019)
- [Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution](https://mlsanity.com/paper/2007.16100/) (2020)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation](https://mlsanity.com/paper/2011.10033/) (2020)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation](https://mlsanity.com/paper/2003.14032/) (2020)
- [SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud](https://mlsanity.com/paper/1710.07368/) (2017)
- [SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaptation for Road-Object Segmentation from a LiDAR Point Cloud](https://mlsanity.com/paper/1809.08495/) (2018)
- [End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds](https://mlsanity.com/paper/1910.06528/) (2019)
- [Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Tracking](https://mlsanity.com/paper/2109.03805/) (2021)
- [(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network](https://mlsanity.com/paper/2102.04530/) (2021)
- [RangeDet:In Defense of Range View for LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2103.10039/) (2021)
- [Pillar-based Object Detection for Autonomous Driving](https://mlsanity.com/paper/2007.10323/) (2020)
- [Seamless Scene Segmentation](https://mlsanity.com/paper/1905.01220/) (2019)
- [Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentation](https://mlsanity.com/paper/2103.14962/) (2021)
- [LiDAR-based Panoptic Segmentation via Dynamic Shifting Network](https://mlsanity.com/paper/2011.11964/) (2020)
- [3D Instance Segmentation via Multi-Task Metric Learning](https://mlsanity.com/paper/1906.08650/) (2019)
- [EfficientLPS: Efficient LiDAR Panoptic Segmentation](https://mlsanity.com/paper/2102.08009/) (2021)
- [What You See is What You Get: Exploiting Visibility for 3D Object Detection](https://mlsanity.com/paper/1912.04986/) (2019)
- [JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds with Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields](https://mlsanity.com/paper/1904.00699/) (2019)

## Built on by (most cited first, 30 of 59)

- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [Rethinking Range View Representation for LiDAR Segmentation](https://mlsanity.com/paper/2303.05367/) (2023)
- [PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentation](https://mlsanity.com/paper/2306.10013/) (2023)
- [LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perception](https://mlsanity.com/paper/2209.09385/) (2022)
- [OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views](https://mlsanity.com/paper/2404.03650/) (2024)
- [UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebase](https://mlsanity.com/paper/2309.05573/) (2023)
- [MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2401.07745/) (2024)
- [FRNet: Frustum-Range Networks for Scalable LiDAR Segmentation](https://mlsanity.com/paper/2312.04484/) (2023)
- [Better Call SAL: Towards Learning to Segment Anything in Lidar](https://mlsanity.com/paper/2403.13129/) (2024)
- [PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness](https://mlsanity.com/paper/2312.02158/) (2023)
- [PIN-SLAM: LiDAR SLAM Using a Point-Based Implicit Neural Representation for Achieving Global Map Consistency](https://mlsanity.com/paper/2401.09101/) (2024)
- [ViewFormer: Exploring Spatiotemporal Modeling for Multi-View 3D Occupancy Perception via View-Guided Transformers](https://mlsanity.com/paper/2405.04299/) (2024)
- [SSC-RS: Elevate LiDAR Semantic Scene Completion with Representation Separation and BEV Fusion](https://mlsanity.com/paper/2306.15349/) (2023)
- [Multi-Space Alignments Towards Universal LiDAR Segmentation](https://mlsanity.com/paper/2405.01538/) (2024)
- [Mask4Former: Mask Transformer for 4D Panoptic Segmentation](https://mlsanity.com/paper/2309.16133/) (2023)
- [PanSt3R: Multi-view Consistent Panoptic Segmentation](https://mlsanity.com/paper/2506.21348/) (2025)
- [4D Panoptic Segmentation as Invariant and Equivariant Field Prediction](https://mlsanity.com/paper/2303.15651/) (2023)
- [LiDAR-Camera Panoptic Segmentation via Geometry-Consistent and Semantic-Aware Alignment](https://mlsanity.com/paper/2308.01686/) (2023)
- [Uncertainty-aware Panoptic Segmentation](https://mlsanity.com/paper/2206.14554/) (2022)
- [Position-Guided Point Cloud Panoptic Segmentation Transformer](https://mlsanity.com/paper/2303.13509/) (2023)
- [Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillation](https://mlsanity.com/paper/2407.13362/) (2024)
- [PUPS: Point Cloud Unified Panoptic Segmentation](https://mlsanity.com/paper/2302.06185/) (2023)
- [Scalable SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2209.08263/) (2022)
- [Walking Your LiDOG: A Journey Through Multiple Domains for LiDAR Semantic Segmentation](https://mlsanity.com/paper/2304.11705/) (2023)
- [3D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillation](https://mlsanity.com/paper/2401.02402/) (2024)
- [4D-Former: Multimodal 4D Panoptic Segmentation](https://mlsanity.com/paper/2311.01520/) (2023)
- [Symbol as Points: Panoptic Symbol Spotting via Point-based Representation](https://mlsanity.com/paper/2401.10556/) (2024)
- [Radar Instance Transformer: Reliable Moving Instance Segmentation in Sparse Radar Point Clouds](https://mlsanity.com/paper/2309.16435/) (2023)
- [Building a Strong Pre-Training Baseline for Universal 3D Large-Scale Perception](https://mlsanity.com/paper/2405.07201/) (2024)
- [Lidar Panoptic Segmentation in an Open World](https://mlsanity.com/paper/2409.14273/) (2024)

## Abstract (arXiv)

As a rising task, panoptic segmentation is faced with challenges in both semantic segmentation and instance segmentation. However, in terms of speed and accuracy, existing LiDAR methods in the field are still limited. In this paper, we propose a fast and high-performance LiDAR-based framework, referred to as Panoptic-PHNet, with three attractive aspects: 1) We introduce a clustering pseudo heatmap as a new paradigm, which, followed by a center grouping module, yields instance centers for efficient clustering without object-level learning tasks. 2) A knn-transformer module is proposed to model the interaction among foreground points for accurate offset regression. 3) For backbone design, we fuse the fine-grained voxel features and the 2D Bird's Eye View (BEV) features with different receptive fields to utilize both detailed and global information. Extensive experiments on both SemanticKITTI dataset and nuScenes dataset show that our Panoptic-PHNet surpasses state-of-the-art methods by remarkable margins with a real-time speed. We achieve the 1st place on the public leaderboard of SemanticKITTI and leading performance on the recently released leaderboard of nuScenes.