# AOP-Net: All-in-One Perception Network for Joint LiDAR-based 3D Object Detection and Panoptic Segmentation

arXiv: https://arxiv.org/abs/2302.00885 · HTML: https://mlsanity.com/paper/2302.00885/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 30 references that are in the map.
- First posted on arXiv 2 Feb 2023.

## Builds on (most cited first, 25 of 25)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [PIXOR: Real-time 3D Object Detection from Point Clouds](https://mlsanity.com/paper/1902.06326/) (2019)
- [Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection](https://mlsanity.com/paper/1908.09492/) (2019)
- [ResMLP: Feedforward networks for image classification with data-efficient training](https://mlsanity.com/paper/2105.03404/) (2021)
- [CycleMLP: A MLP-like Architecture for Dense Prediction](https://mlsanity.com/paper/2107.10224/) (2021)
- [Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentation](https://mlsanity.com/paper/2103.14962/) (2021)
- [FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection](https://mlsanity.com/paper/2106.12449/) (2021)
- [Object as Hotspots: An Anchor-Free 3D Object Detection Approach via Firing of Hotspots](https://mlsanity.com/paper/1912.12791/) (2019)
- [Range Conditioned Dilated Convolutions for Scale Invariant 3D Object Detection](https://mlsanity.com/paper/2005.09927/) (2020)
- [EfficientLPS: Efficient LiDAR Panoptic Segmentation](https://mlsanity.com/paper/2102.08009/) (2021)
- [Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet](https://mlsanity.com/paper/2105.02723/) (2021)
- [MOPT: Multi-Object Panoptic Tracking](https://mlsanity.com/paper/2004.08189/) (2020)
- [Panoster: End-to-end Panoptic Segmentation of LiDAR Point Clouds](https://mlsanity.com/paper/2010.15157/) (2020)
- [GP-S3Net: Graph-based Panoptic Sparse Semantic Segmentation Network](https://mlsanity.com/paper/2108.08401/) (2021)
- [ConvMLP: Hierarchical Convolutional MLPs for Vision](https://mlsanity.com/paper/2109.04454/) (2021)
- [CPSeg: Cluster-free Panoptic Segmentation of 3D LiDAR Point Clouds](https://mlsanity.com/paper/2111.01723/) (2021)
- [A Versatile Multi-View Framework for LiDAR-based 3D Object Detection with Guidance from Panoptic Segmentation](https://mlsanity.com/paper/2203.02133/) (2022)

## Built on by (most cited first, 2 of 2)

- [Panoptic Perception for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.15388/) (2024)
- [PANet: LiDAR Panoptic Segmentation with Sparse Instance Proposal and Aggregation](https://mlsanity.com/paper/2306.15348/) (2023)

## Abstract (arXiv)

LiDAR-based 3D object detection and panoptic segmentation are two crucial tasks in the perception systems of autonomous vehicles and robots. In this paper, we propose All-in-One Perception Network (AOP-Net), a LiDAR-based multi-task framework that combines 3D object detection and panoptic segmentation. In this method, a dual-task 3D backbone is developed to extract both panoptic- and detection-level features from the input LiDAR point cloud. Also, a new 2D backbone that intertwines Multi-Layer Perceptron (MLP) and convolution layers is designed to further improve the detection task performance. Finally, a novel module is proposed to guide the detection head by recovering useful features discarded during down-sampling operations in the 3D backbone. This module leverages estimated instance segmentation masks to recover detailed information from each candidate object. The AOP-Net achieves state-of-the-art performance for published works on the nuScenes benchmark for both 3D object detection and panoptic segmentation tasks. Also, experiments show that our method easily adapts to and significantly improves the performance of any BEV-based 3D object detection method.