# Open-set 3D Object Detection

arXiv: https://arxiv.org/abs/2112.01135 · HTML: https://mlsanity.com/paper/2112.01135/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 33 references that are in the map.
- First posted on arXiv 2 Dec 2021.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [PIXOR: Real-time 3D Object Detection from Point Clouds](https://mlsanity.com/paper/1902.06326/) (2019)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [Deep Autoencoding Models for Unsupervised Anomaly Segmentation in Brain MR Images](https://mlsanity.com/paper/1804.04488/) (2018)
- [Detecting the Unexpected via Image Resynthesis](https://mlsanity.com/paper/1904.07595/) (2019)
- [Robust Classification with Convolutional Prototype Learning](https://mlsanity.com/paper/1805.03438/) (2018)
- [Dropout Sampling for Robust Object Detection in Open-Set Conditions](https://mlsanity.com/paper/1710.06677/) (2017)
- [Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation](https://mlsanity.com/paper/2003.08440/) (2020)
- [Identifying Unknown Instances for Autonomous Driving](https://mlsanity.com/paper/1910.11296/) (2019)
- [Rethinking PointNet Embedding for Faster and Compact Model](https://mlsanity.com/paper/2007.15855/) (2020)

## Built on by (most cited first, 29 of 29)

- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [Open-Vocabulary Point-Cloud Object Detection without 3D Annotation](https://mlsanity.com/paper/2304.00788/) (2023)
- [Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving](https://mlsanity.com/paper/2210.08061/) (2022)
- [Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Driving](https://mlsanity.com/paper/2309.14491/) (2023)
- [End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotation](https://mlsanity.com/paper/2406.17680/) (2024)
- [Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2303.01212/) (2023)
- [Towards Unsupervised Object Detection From LiDAR Point Clouds](https://mlsanity.com/paper/2311.02007/) (2023)
- [OpenSight: A Simple Open-Vocabulary Framework for LiDAR-Based Object Detection](https://mlsanity.com/paper/2312.08876/) (2023)
- [OV-Uni3DETR: Towards Unified Open-Vocabulary 3D Object Detection via Cycle-Modality Propagation](https://mlsanity.com/paper/2403.19580/) (2024)
- [3DOS: Towards 3D Open Set Learning -- Benchmarking and Understanding Semantic Novelty Detection on Point Clouds](https://mlsanity.com/paper/2207.11554/) (2022)
- [PointCaM: Cut-and-Mix for Open-Set Point Cloud Learning](https://mlsanity.com/paper/2212.02011/) (2022)
- [Spotting the Unexpected (STU): A 3D LiDAR Dataset for Anomaly Segmentation in Autonomous Driving](https://mlsanity.com/paper/2505.02148/) (2025)
- [Segmenting Known Objects and Unseen Unknowns without Prior Knowledge](https://mlsanity.com/paper/2209.05407/) (2022)
- [Shelf-Supervised Cross-Modal Pre-Training for 3D Object Detection](https://mlsanity.com/paper/2406.10115/) (2024)
- [Approaching Outside: Scaling Unsupervised 3D Object Detection from 2D Scene](https://mlsanity.com/paper/2407.08569/) (2024)
- [Run-time Introspection of 2D Object Detection in Automated Driving Systems Using Learning Representations](https://mlsanity.com/paper/2403.01172/) (2024)
- [ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Driving](https://mlsanity.com/paper/2411.05311/) (2024)
- [Multimodal Detection of Unknown Objects on Roads for Autonomous Driving](https://mlsanity.com/paper/2205.01414/) (2022)
- [Run-time Monitoring of 3D Object Detection in Automated Driving Systems Using Early Layer Neural Activation Patterns](https://mlsanity.com/paper/2404.07685/) (2024)
- [HD-OOD3D: Supervised and Unsupervised Out-of-Distribution object detection in LiDAR data](https://mlsanity.com/paper/2410.23767/) (2024)
- [MonoSOWA: Scalable monocular 3D Object detector Without human Annotations](https://mlsanity.com/paper/2501.09481/) (2025)
- [Stereo-based 3D Anomaly Object Detection for Autonomous Driving: A New Dataset and Baseline](https://mlsanity.com/paper/2507.09214/) (2025)
- [PartCom: Part Composition Learning for 3D Open-Set Recognition](https://mlsanity.com/paper/2211.10880/) (2022)
- [OpenPatch: a 3D patchwork for Out-Of-Distribution detection](https://mlsanity.com/paper/2310.03388/) (2023)
- [Label-Free Model Failure Detection for Lidar-based Point Cloud Segmentation](https://mlsanity.com/paper/2407.14306/) (2024)
- [DDStereo: Efficient Dual Decoder Transformers for Stereo 3D Road Anomaly Detection](https://mlsanity.com/paper/2606.24805/) (2026)
- [UOD: Unseen Object Detection in 3D Point Cloud](https://mlsanity.com/paper/2401.03846/) (2024)
- [RAD: Retrieval-Augmented Monocular Metric Depth Estimation for Underrepresented Classes](https://mlsanity.com/paper/2602.09532/) (2026)
- [Energy-Based Open-Set Active Learning for Object Classification](https://mlsanity.com/paper/2604.20083/) (2026)

## Abstract (arXiv)

3D object detection has been wildly studied in recent years, especially for robot perception systems. However, existing 3D object detection is under a closed-set condition, meaning that the network can only output boxes of trained classes. Unfortunately, this closed-set condition is not robust enough for practical use, as it will identify unknown objects as known by mistake. Therefore, in this paper, we propose an open-set 3D object detector, which aims to (1) identify known objects, like the closed-set detection, and (2) identify unknown objects and give their accurate bounding boxes. Specifically, we divide the open-set 3D object detection problem into two steps: (1) finding out the regions containing the unknown objects with high probability and (2) enclosing the points of these regions with proper bounding boxes. The first step is solved by the finding that unknown objects are often classified as known objects with low confidence, and we show that the Euclidean distance sum based on metric learning is a better confidence score than the naive softmax probability to differentiate unknown objects from known objects. On this basis, unsupervised clustering is used to refine the bounding boxes of unknown objects. The proposed method combining metric learning and unsupervised clustering is called the MLUC network. Our experiments show that our MLUC network achieves state-of-the-art performance and can identify both known and unknown objects as expected.