mlsanityData to 7 Oct 2026

Open-Vocabulary Point-Cloud Object Detection without 3D Annotation

arXiv 2304.00788 ↗First posted 3 Apr 2023cs.CVcited by 81

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

The goal of open-vocabulary detection is to identify novel objects based on arbitrary textual descriptions. In this paper, we address open-vocabulary 3D point-cloud detection by a dividing-and-conquering strategy, which involves: 1) developing a point-cloud detector that can learn a general representation for localizing various objects, and 2) connecting textual and point-cloud representations to enable the detector to classify novel object categories based on text prompting. Specifically, we resort to rich image pre-trained models, by which the point-cloud detector learns localizing objects under the supervision of predicted 2D bounding boxes from 2D pre-trained detectors. Moreover, we propose a novel de-biased triplet cross-modal contrastive learning to connect the modalities of image, point-cloud and text, thereby enabling the point-cloud detector to benefit from vision-language pre-trained models,i.e.,CLIP. The novel use of image and vision-language pre-trained models for point-cloud detectors allows for open-vocabulary 3D object detection without the need for 3D annotations. Experiments demonstrate that the proposed method improves at least 3.03 points and 7.47 points over a wide range of baselines on the ScanNet and SUN RGB-D datasets, respectively. Furthermore, we provide a comprehensive analysis to explain why our approach works.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 35 references that are in the map. First posted on arXiv 3 Apr 2023.

Builds on

26 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021PointCLIP: Point Cloud Understanding by CLIP4 Dec

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2017Mask R-CNNcited by 7,259
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  10. 2015Fast R-CNNcited by 2,930
  11. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  12. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
13 more
  1. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  2. 2020Debiased Contrastive Learningcited by 354
  3. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  4. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  5. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  6. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  7. 2019Identifying Unknown Instances for Autonomous Drivingcited by 73
  8. 2020Comprehensive Attention Self-Distillation for Weakly-Supervised Object Detectioncited by 50
  9. 2021Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Modelscited by 41
  10. 2021Open-set 3D Object Detectioncited by 29
  11. 2022Open-world Semantic Segmentation for LIDAR Point Cloudscited by 29
  12. 2019Weakly Supervised Object Detection with Segmentation Collaborationcited by 26
  13. 20223DOS: Towards 3D Open Set Learning -- Benchmarking and Understanding Semantic Novelty Detection on Point Cloudscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 24; 2025: 26; 2026: 15.

81 papers.

Sort:

By area:

  1. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205
  2. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  3. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  4. 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 141
  5. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  6. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  7. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  8. 2023Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labelscited by 64
  9. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  10. 2023CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detectioncited by 51
  11. 2023The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained understandingcited by 29
  12. 2023Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detectioncited by 24
38 more
  1. 2023OpenSight: A Simple Open-Vocabulary Framework for LiDAR-Based Object Detectioncited by 22
  2. 2024Open Vocabulary Monocular 3D Object Detectioncited by 22
  3. 2024Find n' Propagate: Open-Vocabulary 3D Object Detection in Urban Environmentscited by 21
  4. 2023A Survey of Label-Efficient Deep Learning for 3D Point Cloudscited by 20
  5. 2024UniDet3D: Multi-dataset Indoor 3D Object Detectioncited by 20
  6. 2023FM-OV3D: Foundation Model-based Cross-modal Knowledge Blending for Open-Vocabulary 3D Detectioncited by 19
  7. 2024OV-Uni3DETR: Towards Unified Open-Vocabulary 3D Object Detection via Cycle-Modality Propagationcited by 19
  8. 2023Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-trainingcited by 17
  9. 2024Collaborative Novel Object Discovery and Box-Guided Cross-Modal Alignment for Open-Vocabulary 3D Object Detectioncited by 17
  10. 2024ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Imagescited by 17
  11. 2024Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understandingcited by 15
  12. 2024Unlocking Textual and Visual Wisdom: Open-Vocabulary 3D Object Detection Enhanced by Comprehensive Guidance from Text and Imagecited by 14
  13. 2023Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluationcited by 12
  14. 2024Global-Local Collaborative Inference with LLM for Lidar-Based Open-Vocabulary Detectioncited by 10
  15. 2024Diorama: Unleashing Zero-shot Single-view 3D Indoor Scene Modelingcited by 10
  16. 2024OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Imagescited by 9
  17. 2024OpenAD: Open-World Autonomous Driving Benchmark for 3D Object Detectioncited by 9
  18. 2024One for All: Multi-Domain Joint Training for Point Cloud Based 3D Object Detectioncited by 8
  19. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sightcited by 8
  20. 2025OpenM3D: Open Vocabulary Multi-view Indoor 3D Object Detection without Human Annotationscited by 7
  21. 2026ParkGaussian: Surround-view 3D Gaussian Splatting for Autonomous Parkingcited by 7
  22. 2024ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Drivingcited by 6
  23. 2026LabelAny3D: Label Any Object 3D in the Wildcited by 6
  24. 2024HaLo-NeRF: Learning Geometry-Guided Semantics for Exploring Unconstrained Photo Collectionscited by 4
  25. 2024V-MIND: Building Versatile Monocular Indoor 3D Detector with Diverse 2D Annotationscited by 4
  26. 2025Point-Cache: Test-time Dynamic and Hierarchical Cache for Robust and Generalizable Point Cloud Analysiscited by 4
  27. 2025UKBOB: One Billion MRI Labeled Masks for Generalizable 3D Medical Image Segmentationcited by 4
  28. 2025Zoo3D: Zero-Shot 3D Object Detection at Scene Levelcited by 4
  29. 2024Vision-Language Guidance for LiDAR-based Unsupervised 3D Object Detectioncited by 3
  30. 2024SA3DIP: Segment Any 3D Instance with Potential 3D Priorscited by 3
  31. 2025GrabS: Generative Embodied Agent for 3D Object Segmentation without Scene Supervisioncited by 3
  32. 2025BoxFusion: Reconstruction-Free Open-Vocabulary 3D Object Detection via Real-Time Multi-View Box Fusioncited by 3
  33. 2025ShelfGaussian: Shelf-Supervised Open-Vocabulary Gaussian-based 3D Scene Understandingcited by 3
  34. 2024Open3DTrack: Towards Open-Vocabulary 3D Multi-Object Trackingcited by 2
  35. 2024Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Datacited by 2
  36. 2025OV-SCAN: Semantically Consistent Alignment for Novel Object Discovery in Open-Vocabulary 3D Object Detectioncited by 2
  37. 2025InstDrive: Instance-Aware 3D Gaussian Splatting for Driving Scenescited by 2
  38. 2025Towards Open-Vocabulary Multimodal 3D Object Detection with Attributescited by 2

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SODA: Out-of-Distribution Detection in Domain-Shifted Point Clouds via Neighborhood Propagation
  2. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data
  3. 2023Joint Representation Learning for Text and 3D Point Cloud
  4. 2025Occlusion-aware Text-Image-Point Cloud Pretraining for Open-World 3D Object Recognition
  5. 2025Exploiting Vision Language Model for Training-Free 3D Point Cloud OOD Detection via Graph Score Propagation
  6. 2025A Review of 3D Object Detection with Vision-Language Models
  7. 2025Hierarchical Cross-Modal Alignment for Open-Vocabulary 3D Object Detection
  8. 2023What Makes Good Open-Vocabulary Detector: A Disassembling Perspective