# Point-DETR3D: Leveraging Imagery Data with Spatial Point Prior for Weakly Semi-supervised 3D Object Detection

arXiv: https://arxiv.org/abs/2403.15317 · HTML: https://mlsanity.com/paper/2403.15317/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework](https://mlsanity.com/paper/2205.13790/) (2022)
- [Embracing Single Stride 3D Object Detector with Sparse Transformer](https://mlsanity.com/paper/2112.06375/) (2021)
- [Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation](https://mlsanity.com/paper/1502.02734/) (2015)
- [Weakly Supervised Deep Detection Networks](https://mlsanity.com/paper/1511.02853/) (2015)
- [FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stochastic Inference](https://mlsanity.com/paper/1902.10421/) (2019)
- [DeepInteraction: 3D Object Detection via Modality Interaction](https://mlsanity.com/paper/2208.11112/) (2022)
- [Multiple Instance Detection Network with Online Instance Classifier Refinement](https://mlsanity.com/paper/1704.00138/) (2017)
- [SESS: Self-Ensembling Semi-Supervised 3D Object Detection](https://mlsanity.com/paper/1912.11803/) (2019)
- [3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detection](https://mlsanity.com/paper/2012.04355/) (2020)
- [Weakly Supervised 3D Object Detection from Lidar Point Cloud](https://mlsanity.com/paper/2007.11901/) (2020)
- [Points as Queries: Weakly Semi-supervised Object Detection by Points](https://mlsanity.com/paper/2104.07434/) (2021)
- [Object DGCNN: 3D Object Detection using Dynamic Graphs](https://mlsanity.com/paper/2110.06923/) (2021)
- [Semi-supervised 3D Object Detection with Proficient Teachers](https://mlsanity.com/paper/2207.12655/) (2022)
- [Weakly- and Semi-Supervised Panoptic Segmentation](https://mlsanity.com/paper/1808.03575/) (2018)
- [Towards Precise End-to-end Weakly Supervised Object Detection Network](https://mlsanity.com/paper/1911.12148/) (2019)
- [Group R-CNN for Weakly Semi-supervised Object Detection with Points](https://mlsanity.com/paper/2205.05920/) (2022)
- [The Devil is in the Points: Weakly Semi-Supervised Instance Segmentation via Point-Guided Mask Representation](https://mlsanity.com/paper/2303.15062/) (2023)
- [Back to Reality: Weakly-supervised 3D Object Detection with Shape-guided Label Enhancement](https://mlsanity.com/paper/2203.05238/) (2022)
- [Point-Teaching: Weakly Semi-Supervised Object Detection with Point Annotations](https://mlsanity.com/paper/2206.00274/) (2022)

## Built on by (most cited first, 3 of 3)

- [LiDAR Remote Sensing Meets Weak Supervision: Concepts, Methods, and Perspectives](https://mlsanity.com/paper/2503.18384/) (2025)
- [UAVDB: Point-Guided Masks for UAV Detection and Segmentation](https://mlsanity.com/paper/2409.06490/) (2024)
- [MVAT: Multi-View Aware Teacher for Weakly Supervised 3D Object Detection](https://mlsanity.com/paper/2509.07507/) (2025)

## Abstract (arXiv)

Training high-accuracy 3D detectors necessitates massive labeled 3D annotations with 7 degree-of-freedom, which is laborious and time-consuming. Therefore, the form of point annotations is proposed to offer significant prospects for practical applications in 3D detection, which is not only more accessible and less expensive but also provides strong spatial information for object localization. In this paper, we empirically discover that it is non-trivial to merely adapt Point-DETR to its 3D form, encountering two main bottlenecks: 1) it fails to encode strong 3D prior into the model, and 2) it generates low-quality pseudo labels in distant regions due to the extreme sparsity of LiDAR points. To overcome these challenges, we introduce Point-DETR3D, a teacher-student framework for weakly semi-supervised 3D detection, designed to fully capitalize on point-wise supervision within a constrained instance-wise annotation budget.Different from Point-DETR which encodes 3D positional information solely through a point encoder, we propose an explicit positional query initialization strategy to enhance the positional prior. Considering the low quality of pseudo labels at distant regions produced by the teacher model, we enhance the detector's perception by incorporating dense imagery data through a novel Cross-Modal Deformable RoI Fusion (D-RoI).Moreover, an innovative point-guided self-supervised learning technique is proposed to allow for fully exploiting point priors, even in student models.Extensive experiments on representative nuScenes dataset demonstrate our Point-DETR3D obtains significant improvements compared to previous works. Notably, with only 5% of labeled data, Point-DETR3D achieves over 90% performance of its fully supervised counterpart.