mlsanityData to 7 Oct 2026

Open-Vocabulary 3D Detection via Image-level Class and Debiased Cross-modal Contrastive Learning

arXiv 2207.01987 ↗First posted 5 Jul 2022cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

Current point-cloud detection methods have difficulty detecting the open-vocabulary objects in the real world, due to their limited generalization capability. Moreover, it is extremely laborious and expensive to collect and fully annotate a point-cloud detection dataset with numerous classes of objects, leading to the limited classes of existing point-cloud datasets and hindering the model to learn general representations to achieve open-vocabulary point-cloud detection. As far as we know, we are the first to study the problem of open-vocabulary 3D point-cloud detection. Instead of seeking a point-cloud dataset with full labels, we resort to ImageNet1K to broaden the vocabulary of the point-cloud detector. We propose OV-3DETIC, an Open-Vocabulary 3D DETector using Image-level Class supervision. Specifically, we take advantage of two modalities, the image modality for recognition and the point-cloud modality for localization, to generate pseudo labels for unseen classes. Then we propose a novel debiased cross-modal contrastive learning method to transfer the knowledge from image modality to point-cloud modality during training. Without hurting the latency during inference, OV-3DETIC makes the point-cloud detector capable of achieving open-vocabulary detection. Extensive experiments demonstrate that the proposed OV-3DETIC achieves at least 10.77 % mAP improvement (absolute value) and 9.56 % mAP improvement (absolute value) by a wide range of baselines on the SUN-RGBD dataset and ScanNet dataset, respectively. Besides, we conduct sufficient experiments to shed light on why the proposed OV-3DETIC works.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 40 of its 48 references that are in the map. First posted on arXiv 5 Jul 2022.

Builds on

40 of 48 references.

References · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2020Point Transformercited by 1,462
  10. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  11. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  12. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 890
28 more
  1. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 721
  2. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  3. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  4. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  5. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  6. 2019From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Networkcited by 366
  7. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  8. 2020Debiased Contrastive Learningcited by 354
  9. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  10. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  11. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  12. 2017SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloudcited by 231
  13. 2018SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaptation for Road-Object Segmentation from a LiDAR Point Cloudcited by 228
  14. 2021Group-Free 3D Object Detection via Transformerscited by 215
  15. 2018Zero-Shot Object Detectioncited by 189
  16. 2017PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimationcited by 170
  17. 2020SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentationcited by 170
  18. 2020H3DNet: 3D Object Detection Using Hybrid Geometric Primitivescited by 123
  19. 2019Transferable Contrastive Network for Generalized Zero-Shot Learningcited by 64
  20. 2021Pri3D: Can 3D Priors Help 2D Representation Learning?cited by 61
  21. 2021Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Cloudscited by 55
  22. 2021Zero-Shot Learning on 3D Point Cloud Objects and Beyondcited by 38
  23. 2021Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillationcited by 35
  24. 2019Zero-shot Learning of 3D Point Cloud Objectscited by 34
  25. 2021Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretrainingcited by 32
  26. 2019Mitigating the Hubness Problem for Zero-Shot Learning of 3D Objectscited by 24
  27. 2022DetMatch: Two Teachers are Better Than One for Joint 2D and 3D Semi-Supervised Object Detectioncited by 20
  28. 2021You Only Group Once: Efficient Point-Cloud Processing with Token Representation and Relation Inference Modulecited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 9; 2024: 3; 2025: 6; 2026: 3.

23 papers.

Sort:

By area:

  1. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  2. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  3. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  4. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  5. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  6. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  7. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  8. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69
  9. 2023Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Drivingcited by 37
  10. 2023Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detectioncited by 24
  11. 2024Open Vocabulary Monocular 3D Object Detectioncited by 22
  12. 2023FM-OV3D: Foundation Model-based Cross-modal Knowledge Blending for Open-Vocabulary 3D Detectioncited by 19
11 more
  1. 2024ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Imagescited by 17
  2. 2024Global-Local Collaborative Inference with LLM for Lidar-Based Open-Vocabulary Detectioncited by 10
  3. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sightcited by 8
  4. 2026LabelAny3D: Label Any Object 3D in the Wildcited by 6
  5. 2025Zoo3D: Zero-Shot 3D Object Detection at Scene Levelcited by 4
  6. 2025Towards Open-Vocabulary Multimodal 3D Object Detection with Attributescited by 2
  7. 2026OnlineSI: Taming Large Language Model for Online 3D Understanding and Groundingcited by 2
  8. 2025GLRD: Global-Local Collaborative Reason and Debate with PSL for 3D Open-Vocabulary Detectioncited by 1
  9. 2025Vision Generalist Model: A Surveycited by 0
  10. 2025Sparse Multiview Open-Vocabulary 3D Detectioncited by 0
  11. 2026Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraintscited by 0