# Image-Guided Semantic Pseudo-LiDAR Point Generation for 3D Object Detection

arXiv: https://arxiv.org/abs/2409.14985 · HTML: https://mlsanity.com/paper/2409.14985/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 47 of its 66 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection](https://mlsanity.com/paper/2012.15712/) (2020)
- [PCN: Point Completion Network](https://mlsanity.com/paper/1808.00671/) (2018)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection](https://mlsanity.com/paper/2102.00463/) (2021)
- [DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2203.08195/) (2022)
- [Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving](https://mlsanity.com/paper/1906.06310/) (2019)
- [3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection](https://mlsanity.com/paper/2004.12636/) (2020)
- [PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation](https://mlsanity.com/paper/1711.10871/) (2017)
- [Multimodal Virtual Point 3D Detection](https://mlsanity.com/paper/2111.06881/) (2021)
- [MVX-Net: Multimodal VoxelNet for 3D Object Detection](https://mlsanity.com/paper/1904.01649/) (2019)
- [Focal Sparse Convolutional Networks for 3D Object Detection](https://mlsanity.com/paper/2204.12463/) (2022)
- [SPG: Unsupervised Domain Adaptation for 3D Object Detection via Semantic Point Generation](https://mlsanity.com/paper/2108.06709/) (2021)

## Abstract (arXiv)

In autonomous driving scenarios, accurate perception is becoming an even more critical task for safe navigation. While LiDAR provides precise spatial data, its inherent sparsity makes it difficult to detect small or distant objects. Existing methods try to address this by generating additional points within a Region of Interest (RoI), but relying on LiDAR alone often leads to false positives and a failure to recover meaningful structures. To address these limitations, we propose Image-Guided Semantic Pseudo-LiDAR Point Generation model, called ImagePG, a novel framework that leverages rich RGB image features to generate dense and semantically meaningful 3D points. Our framework includes an Image-Guided RoI Points Generation (IG-RPG) module, which creates pseudo-points guided by image features, and an Image-Aware Occupancy Prediction Network (I-OPN), which provides spatial priors to guide point placement. A multi-stage refinement (MR) module further enhances point quality and detection robustness. To the best of our knowledge, ImagePG is the first method to directly leverage image features for point generation. Extensive experiments on the KITTI and Waymo datasets demonstrate that ImagePG significantly improves the detection of small and distant objects like pedestrians and cyclists, reducing false positives by nearly 50%. On the KITTI benchmark, our framework improves mAP by +1.38%p (car), +7.91%p (pedestrian), and +5.21%p (cyclist) on the test set over the baseline, achieving state-of-the-art cyclist performance on the KITTI leaderboard. The code is available at: https://github.com/MS-LIMA/ImagePG