# LidarMultiNet: Unifying LiDAR Semantic Segmentation, 3D Object Detection, and Panoptic Segmentation in a Single Multi-task Network

arXiv: https://arxiv.org/abs/2206.11428 · HTML: https://mlsanity.com/paper/2206.11428/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 23 Jun 2022.

## Builds on (most cited first, 30 of 30)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation](https://mlsanity.com/paper/2011.10033/) (2020)
- [PIXOR: Real-time 3D Object Detection from Point Clouds](https://mlsanity.com/paper/1902.06326/) (2019)
- [PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation](https://mlsanity.com/paper/2003.14032/) (2020)
- [RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation](https://mlsanity.com/paper/2103.12978/) (2021)
- [Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Tracking](https://mlsanity.com/paper/2109.03805/) (2021)
- [(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network](https://mlsanity.com/paper/2102.04530/) (2021)
- [A2-Nets: Double Attention Networks](https://mlsanity.com/paper/1810.11579/) (2018)
- [LiDAR R-CNN: An Efficient and Universal 3D Object Detector](https://mlsanity.com/paper/2103.15297/) (2021)
- [Improving 3D Object Detection with Channel-wise Transformer](https://mlsanity.com/paper/2108.10723/) (2021)
- [Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentation](https://mlsanity.com/paper/2103.14962/) (2021)
- [MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving](https://mlsanity.com/paper/1612.07695/) (2016)
- [LiDAR-based Panoptic Segmentation via Dynamic Shifting Network](https://mlsanity.com/paper/2011.11964/) (2020)
- [ACFNet: Attentional Class Feature Network for Semantic Segmentation](https://mlsanity.com/paper/1909.09408/) (2019)
- [GP-S3Net: Graph-based Panoptic Sparse Semantic Segmentation Network](https://mlsanity.com/paper/2108.08401/) (2021)
- [DRINet: A Dual-Representation Iterative Learning Network for Point Cloud Segmentation](https://mlsanity.com/paper/2108.04023/) (2021)
- [A 3D Coarse-to-Fine Framework for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1712.00201/) (2017)
- [Iterative Instance Segmentation](https://mlsanity.com/paper/1511.08498/) (2015)
- [A Simple and Efficient Multi-task Network for 3D Object Detection and Road Understanding](https://mlsanity.com/paper/2103.04056/) (2021)

## Built on by (most cited first, 8 of 8)

- [LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perception](https://mlsanity.com/paper/2209.09385/) (2022)
- [FlatFormer: Flattened Window Attention for Efficient Point Cloud Transformer](https://mlsanity.com/paper/2301.08739/) (2023)
- [TorchSparse++: Efficient Training and Inference Framework for Sparse Convolution on GPUs](https://mlsanity.com/paper/2311.12862/) (2023)
- [Once Detected, Never Lost: Surpassing Human Performance in Offline LiDAR based 3D Object Detection](https://mlsanity.com/paper/2304.12315/) (2023)
- [PVT-SSD: Single-Stage 3D Object Detector with Point-Voxel Transformer](https://mlsanity.com/paper/2305.06621/) (2023)
- [RAPiD-Seg: Range-Aware Pointwise Distance Distribution Networks for 3D LiDAR Segmentation](https://mlsanity.com/paper/2407.10159/) (2024)
- [NUC-Net: Non-uniform Cylindrical Partition Network for Efficient LiDAR Semantic Segmentation](https://mlsanity.com/paper/2505.24634/) (2025)
- [RadarFormer: Lightweight and Accurate Real-Time Radar Object Detection Model](https://mlsanity.com/paper/2304.08447/) (2023)

## Abstract (arXiv)

This technical report presents the 1st place winning solution for the Waymo Open Dataset 3D semantic segmentation challenge 2022. Our network, termed LidarMultiNet, unifies the major LiDAR perception tasks such as 3D semantic segmentation, object detection, and panoptic segmentation in a single framework. At the core of LidarMultiNet is a strong 3D voxel-based encoder-decoder network with a novel Global Context Pooling (GCP) module extracting global contextual features from a LiDAR frame to complement its local features. An optional second stage is proposed to refine the first-stage segmentation or generate accurate panoptic segmentation results. Our solution achieves a mIoU of 71.13 and is the best for most of the 22 classes on the Waymo 3D semantic segmentation test set, outperforming all the other 3D semantic segmentation methods on the official leaderboard. We demonstrate for the first time that major LiDAR perception tasks can be unified in a single strong network that can be trained end-to-end.