# Semantics-aware LiDAR-Only Pseudo Point Cloud Generation for 3D Object Detection

arXiv: https://arxiv.org/abs/2309.08932 · HTML: https://mlsanity.com/paper/2309.08932/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 38 references that are in the map.
- First posted on arXiv 16 Sep 2023.

## Builds on (most cited first, 30 of 33)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection](https://mlsanity.com/paper/2012.15712/) (2020)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [STD: Sparse-to-Dense 3D Object Detector for Point Cloud](https://mlsanity.com/paper/1907.10471/) (2019)
- [SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving](https://mlsanity.com/paper/2003.03653/) (2020)
- [Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving](https://mlsanity.com/paper/1906.06310/) (2019)
- [Multimodal Virtual Point 3D Detection](https://mlsanity.com/paper/2111.06881/) (2021)
- [EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection](https://mlsanity.com/paper/2007.08856/) (2020)
- [Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds](https://mlsanity.com/paper/2203.11139/) (2022)
- [IoU Loss for 2D/3D Object Detection](https://mlsanity.com/paper/1908.03851/) (2019)
- [SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloud](https://mlsanity.com/paper/2104.09804/) (2021)
- [Rethinking Pseudo-LiDAR Representation](https://mlsanity.com/paper/2008.04582/) (2020)
- [Improving 3D Object Detection with Channel-wise Transformer](https://mlsanity.com/paper/2108.10723/) (2021)
- [FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection](https://mlsanity.com/paper/2106.12449/) (2021)
- [Sparse Fuse Dense: Towards High Quality 3D Detection with Depth Completion](https://mlsanity.com/paper/2203.09780/) (2022)
- [Virtual Sparse Convolution for Multimodal 3D Object Detection](https://mlsanity.com/paper/2303.02314/) (2023)
- [Pyramid R-CNN: Towards Better Performance and Adaptability for 3D Object Detection](https://mlsanity.com/paper/2109.02499/) (2021)
- [Behind the Curtain: Learning Occluded Shapes for 3D Object Detection](https://mlsanity.com/paper/2112.02205/) (2021)
- [Graph R-CNN: Towards Accurate 3D Object Detection with Semantic-Decorated Local Graph](https://mlsanity.com/paper/2208.03624/) (2022)
- [SDNet: Semantically Guided Depth Estimation Network](https://mlsanity.com/paper/1907.10659/) (2019)

## Abstract (arXiv)

Although LiDAR sensors are crucial for autonomous systems due to providing precise depth information, they struggle with capturing fine object details, especially at a distance, due to sparse and non-uniform data. Recent advances introduced pseudo-LiDAR, i.e., synthetic dense point clouds, using additional modalities such as cameras to enhance 3D object detection. We present a novel LiDAR-only framework that augments raw scans with denser pseudo point clouds by solely relying on LiDAR sensors and scene semantics, omitting the need for cameras. Our framework first utilizes a segmentation model to extract scene semantics from raw point clouds, and then employs a multi-modal domain translator to generate synthetic image segments and depth cues without real cameras. This yields a dense pseudo point cloud enriched with semantic information. We also introduce a new semantically guided projection method, which enhances detection performance by retaining only relevant pseudo points. We applied our framework to different advanced 3D object detection methods and reported up to 2.9% performance upgrade. We also obtained comparable results on the KITTI 3D object detection dataset, in contrast to other state-of-the-art LiDAR-only detectors.