# Behind the Curtain: Learning Occluded Shapes for 3D Object Detection

arXiv: https://arxiv.org/abs/2112.02205 · HTML: https://mlsanity.com/paper/2112.02205/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 54 references that are in the map.
- First posted on arXiv 4 Dec 2021.

## Builds on (most cited first, 30 of 44)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection](https://mlsanity.com/paper/2012.15712/) (2020)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud](https://mlsanity.com/paper/2003.01251/) (2020)
- [Joint 3D Proposal Generation and Object Detection from View Aggregation](https://mlsanity.com/paper/1712.02294/) (2017)
- [STD: Sparse-to-Dense 3D Object Detector for Point Cloud](https://mlsanity.com/paper/1907.10471/) (2019)
- [Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion](https://mlsanity.com/paper/2012.03762/) (2020)
- [Multi-Task Multi-Sensor Fusion for 3D Object Detection](https://mlsanity.com/paper/2012.12397/) (2020)
- [Acquisition of Localization Confidence for Accurate Object Detection](https://mlsanity.com/paper/1807.11590/) (2018)
- [3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection](https://mlsanity.com/paper/2004.12636/) (2020)
- [End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds](https://mlsanity.com/paper/1910.06528/) (2019)
- [EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection](https://mlsanity.com/paper/2007.08856/) (2020)
- [CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection](https://mlsanity.com/paper/2009.00784/) (2020)
- [Pillar-based Object Detection for Autonomous Driving](https://mlsanity.com/paper/2007.10323/) (2020)
- [IoU Loss for 2D/3D Object Detection](https://mlsanity.com/paper/1908.03851/) (2019)
- [Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection](https://mlsanity.com/paper/1903.01864/) (2019)
- [Fast Point R-CNN](https://mlsanity.com/paper/1908.02990/) (2019)
- [TANet: Robust 3D Object Detection from Point Clouds with Triple Attention](https://mlsanity.com/paper/1912.05163/) (2019)
- [CIA-SSD: Confident IoU-Aware Single-Stage Object Detector From Point Cloud](https://mlsanity.com/paper/2012.03015/) (2020)
- [GS3D: An Efficient 3D Object Detection Framework for Autonomous Driving](https://mlsanity.com/paper/1903.10955/) (2019)
- [Grid-GCN for Fast and Scalable Point Cloud Learning](https://mlsanity.com/paper/1912.02984/) (2019)

## Built on by (most cited first, 30 of 58)

- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [CenterFormer: Center-based Transformer for 3D Object Detection](https://mlsanity.com/paper/2209.05588/) (2022)
- [Virtual Sparse Convolution for Multimodal 3D Object Detection](https://mlsanity.com/paper/2303.02314/) (2023)
- [Transformation-Equivariant 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/2211.11962/) (2022)
- [MonoCD: Monocular 3D Object Detection with Complementary Depths](https://mlsanity.com/paper/2404.03181/) (2024)
- [Uni3DETR: Unified 3D Detection Transformer](https://mlsanity.com/paper/2310.05699/) (2023)
- [Object Detection with Transformers: A Review](https://mlsanity.com/paper/2306.04670/) (2023)
- [MonoDGP: Monocular 3D Object Detection with Decoupled-Query and Geometry-Error Priors](https://mlsanity.com/paper/2410.19590/) (2024)
- [PVT-SSD: Single-Stage 3D Object Detector with Point-Voxel Transformer](https://mlsanity.com/paper/2305.06621/) (2023)
- [MAELi: Masked Autoencoder for Large-Scale LiDAR Point Clouds](https://mlsanity.com/paper/2212.07207/) (2022)
- [Rethinking IoU-based Optimization for Single-stage 3D Object Detection](https://mlsanity.com/paper/2207.09332/) (2022)
- [Dense Voxel Fusion for 3D Object Detection](https://mlsanity.com/paper/2203.00871/) (2022)
- [MonoMAE: Enhancing Monocular 3D Detection through Depth-Aware Masked Autoencoders](https://mlsanity.com/paper/2405.07696/) (2024)
- [Sparse2Dense: Learning to Densify 3D Features for 3D Object Detection](https://mlsanity.com/paper/2211.13067/) (2022)
- [PointSSC: A Cooperative Vehicle-Infrastructure Point Cloud Benchmark for Semantic Scene Completion](https://mlsanity.com/paper/2309.12708/) (2023)
- [Unsupervised Adaptation from Repeated Traversals for Autonomous Driving](https://mlsanity.com/paper/2303.15286/) (2023)
- [Curricular Object Manipulation in LiDAR-based Object Detection](https://mlsanity.com/paper/2304.04248/) (2023)
- [PG-RCNN: Semantic Surface Point Generation for 3D Object Detection](https://mlsanity.com/paper/2307.12637/) (2023)
- [Real-Time and Robust 3D Object Detection Within Road-Side LiDARs Using Domain Adaptation](https://mlsanity.com/paper/2204.00132/) (2022)
- [Multimodal Transformer for Automatic 3D Annotation and Object Detection](https://mlsanity.com/paper/2207.09805/) (2022)
- [An Empirical Study of the Generalization Ability of Lidar 3D Object Detectors to Unseen Domains](https://mlsanity.com/paper/2402.17562/) (2024)
- [Common Corruption Robustness of Point Cloud Detectors: Benchmark and Enhancement](https://mlsanity.com/paper/2210.05896/) (2022)
- [SSD-MonoDETR: Supervised Scale-aware Deformable Transformer for Monocular 3D Object Detection](https://mlsanity.com/paper/2305.07270/) (2023)
- [Paint and Distill: Boosting 3D Object Detection with Semantic Passing Network](https://mlsanity.com/paper/2207.05497/) (2022)
- [CT3D++: Improving 3D Object Detection with Keypoint-induced Channel-wise Transformer](https://mlsanity.com/paper/2406.08152/) (2024)
- [Generalized Few-Shot 3D Object Detection of LiDAR Point Cloud for Autonomous Driving](https://mlsanity.com/paper/2302.03914/) (2023)
- [Domain Generalization of 3D Object Detection by Density-Resampling](https://mlsanity.com/paper/2311.10845/) (2023)
- [OBMO: One Bounding Box Multiple Objects for Monocular 3D Object Detection](https://mlsanity.com/paper/2212.10049/) (2022)
- [Weak-to-Strong 3D Object Detection with X-Ray Distillation](https://mlsanity.com/paper/2404.00679/) (2024)
- [Multi-Modal 3D Object Detection by Box Matching](https://mlsanity.com/paper/2305.07713/) (2023)

## Abstract (arXiv)

Advances in LiDAR sensors provide rich 3D data that supports 3D scene understanding. However, due to occlusion and signal miss, LiDAR point clouds are in practice 2.5D as they cover only partial underlying shapes, which poses a fundamental challenge to 3D perception. To tackle the challenge, we present a novel LiDAR-based 3D object detection model, dubbed Behind the Curtain Detector (BtcDet), which learns the object shape priors and estimates the complete object shapes that are partially occluded (curtained) in point clouds. BtcDet first identifies the regions that are affected by occlusion and signal miss. In these regions, our model predicts the probability of occupancy that indicates if a region contains object shapes. Integrated with this probability map, BtcDet can generate high-quality 3D proposals. Finally, the probability of occupancy is also integrated into a proposal refinement module to generate the final bounding boxes. Extensive experiments on the KITTI Dataset and the Waymo Open Dataset demonstrate the effectiveness of BtcDet. Particularly, for the 3D detection of both cars and cyclists on the KITTI benchmark, BtcDet surpasses all of the published state-of-the-art methods by remarkable margins. Code is released (https://github.com/Xharlie/BtcDet}{https://github.com/Xharlie/BtcDet).