# A Simple and Efficient Multi-task Network for 3D Object Detection and Road Understanding

arXiv: https://arxiv.org/abs/2103.04056 · HTML: https://mlsanity.com/paper/2103.04056/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 6 Mar 2021.

## Builds on (most cited first, 25 of 25)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [PIXOR: Real-time 3D Object Detection from Point Clouds](https://mlsanity.com/paper/1902.06326/) (2019)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [Joint 3D Proposal Generation and Object Detection from View Aggregation](https://mlsanity.com/paper/1712.02294/) (2017)
- [Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1706.01307/) (2017)
- [Multi-Task Multi-Sensor Fusion for 3D Object Detection](https://mlsanity.com/paper/2012.12397/) (2020)
- [Predicting Semantic Map Representations from Images using Pyramid Occupancy Networks](https://mlsanity.com/paper/2003.13402/) (2020)
- [PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation](https://mlsanity.com/paper/1711.10871/) (2017)
- [SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation](https://mlsanity.com/paper/2004.01803/) (2020)
- [MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving](https://mlsanity.com/paper/1612.07695/) (2016)
- [HDNET: Exploiting HD Maps for 3D Object Detection](https://mlsanity.com/paper/2012.11704/) (2020)
- [SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentation for Accurate Freespace Detection](https://mlsanity.com/paper/2008.11351/) (2020)
- [Convolutional Recurrent Network for Road Boundary Extraction](https://mlsanity.com/paper/2012.12160/) (2020)
- [Deep Multi-Sensor Lane Detection](https://mlsanity.com/paper/1905.01555/) (2019)
- [Labels Are Not Perfect: Inferring Spatial Uncertainty in Object Detection](https://mlsanity.com/paper/2012.12195/) (2020)
- [UrbanLoco: A Full Sensor Suite Dataset for Mapping and Localization in Urban Scenes](https://mlsanity.com/paper/1912.09513/) (2019)
- [Driving Scene Perception Network: Real-time Joint Detection, Depth Estimation and Semantic Segmentation](https://mlsanity.com/paper/1803.03778/) (2018)

## Built on by (most cited first, 17 of 17)

- [LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perception](https://mlsanity.com/paper/2209.09385/) (2022)
- [Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models](https://mlsanity.com/paper/2106.04180/) (2021)
- [SA-BEV: Generating Semantic-Aware Bird's-Eye-View Feature for Multi-view 3D Object Detection](https://mlsanity.com/paper/2307.11477/) (2023)
- [DetMatch: Two Teachers are Better Than One for Joint 2D and 3D Semi-Supervised Object Detection](https://mlsanity.com/paper/2203.09510/) (2022)
- [FULLER: Unified Multi-modality Multi-task 3D Perception via Multi-level Gradient Calibration](https://mlsanity.com/paper/2307.16617/) (2023)
- [LiDARFormer: A Unified Transformer-based Multi-task Network for LiDAR Perception](https://mlsanity.com/paper/2303.12194/) (2023)
- [M3Net: Multimodal Multi-task Learning for 3D Detection, Segmentation, and Occupancy Prediction in Autonomous Driving](https://mlsanity.com/paper/2503.18100/) (2025)
- [A Point-Based Approach to Efficient LiDAR Multi-Task Perception](https://mlsanity.com/paper/2404.12798/) (2024)
- [Cross-Task Attack: A Self-Supervision Generative Framework Based on Attention Shift](https://mlsanity.com/paper/2407.13700/) (2024)
- [Advancing 3D Point Cloud Understanding through Deep Transfer Learning: A Comprehensive Survey](https://mlsanity.com/paper/2407.17877/) (2024)
- [LidarMultiNet: Unifying LiDAR Semantic Segmentation, 3D Object Detection, and Panoptic Segmentation in a Single Multi-task Network](https://mlsanity.com/paper/2206.11428/) (2022)
- [UniVision: A Unified Framework for Vision-Centric 3D Perception](https://mlsanity.com/paper/2401.06994/) (2024)
- [Panoptic Perception for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.15388/) (2024)
- [LiDAR-BEVMTN: Real-Time LiDAR Bird's-Eye View Multi-Task Perception Network for Autonomous Driving](https://mlsanity.com/paper/2307.08850/) (2023)
- [LiSD: An Efficient Multi-Task Learning Framework for LiDAR Segmentation and Detection](https://mlsanity.com/paper/2406.07023/) (2024)
- [Instance-aware 3D Semantic Segmentation powered by Shape Generators and Classifiers](https://mlsanity.com/paper/2311.12291/) (2023)
- [3D Object Visibility Prediction in Autonomous Driving](https://mlsanity.com/paper/2403.03681/) (2024)

## Abstract (arXiv)

Detecting dynamic objects and predicting static road information such as drivable areas and ground heights are crucial for safe autonomous driving. Previous works studied each perception task separately, and lacked a collective quantitative analysis. In this work, we show that it is possible to perform all perception tasks via a simple and efficient multi-task network. Our proposed network, LidarMTL, takes raw LiDAR point cloud as inputs, and predicts six perception outputs for 3D object detection and road understanding. The network is based on an encoder-decoder architecture with 3D sparse convolution and deconvolution operations. Extensive experiments verify the proposed method with competitive accuracies compared to state-of-the-art object detectors and other task-specific networks. LidarMTL is also leveraged for online localization. Code and pre-trained model have been made available at https://github.com/frankfengdi/LidarMTL.