mlsanityData to 7 Oct 2026

OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Data

arXiv 2311.02873 ↗First posted 6 Nov 2023cs.CV cs.ROcited by 81

Not on the map. Why?

Abstract (arXiv)

This work presents OVIR-3D, a straightforward yet effective method for open-vocabulary 3D object instance retrieval without using any 3D data for training. Given a language query, the proposed method is able to return a ranked set of 3D object instance segments based on the feature similarity of the instance and the text query. This is achieved by a multi-view fusion of text-aligned 2D region proposals into 3D space, where the 2D region proposal network could leverage 2D datasets, which are more accessible and typically larger than 3D datasets. The proposed fusion process is efficient as it can be performed in real-time for most indoor 3D scenes and does not require additional training in 3D space. Experiments on public datasets and a real robot show the effectiveness of the method and its potential for applications in robot navigation and manipulation.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 37 of its 51 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

37 of 51 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2017Mask R-CNNcited by 7,259
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  10. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  11. 2021Grounded Language-Image Pre-trainingcited by 1,147
  12. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
25 more
  1. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  2. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  3. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  4. 2022Language-driven Semantic Segmentationcited by 711
  5. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  6. 2021NICE-SLAM: Neural Implicit Scalable Encoding for SLAMcited by 586
  7. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  8. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  9. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  10. 2023LERF: Language Embedded Radiance Fieldscited by 538
  11. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  12. 2021iMAP: Implicit Mapping and Positioning in Real-Timecited by 476
  13. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  14. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  15. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  16. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  17. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  18. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  19. 2016SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networkscited by 134
  20. 2018Fusion++: Volumetric Object-Level SLAMcited by 97
  21. 2019PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Thingscited by 97
  22. 2018MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objectscited by 95
  23. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  24. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  25. 2023vMAP: Vectorised Object Mapping for Neural Field SLAMcited by 67

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 32; 2025: 32; 2026: 14.

81 papers.

Sort:

By area:

  1. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  2. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  3. 2025PARTFIELD: Learning 3D Feature Fields for Part Segmentation and Beyondcited by 91
  4. 2023SAI3D: Segment Any Instance in 3D Scenescited by 90
  5. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  6. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  7. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  8. 2024MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentationcited by 56
  9. 2024GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary Semantic-space Hyperplanecited by 53
  10. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  11. 2024EgoLifter: Open-world 3D Segmentation for Egocentric Perceptioncited by 46
  12. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
38 more
  1. 2023SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentationcited by 44
  2. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  3. 2024EmbodiedSAM: Online Segment Any 3D Thing in Real Timecited by 40
  4. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  5. 2024Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentationcited by 30
  6. 2025ILIAS: Instance-Level Image retrieval At Scalecited by 29
  7. 2024OpenGraph: Open-Vocabulary Hierarchical 3D Graph Representation in Large-Scale Outdoor Environmentscited by 22
  8. 2024VEON: Vocabulary-Enhanced Occupancy Predictioncited by 22
  9. 2024ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Imagescited by 17
  10. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentationcited by 16
  11. 2024Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Trackingcited by 14
  12. 2025AutoOcc: Automatic Open-Ended Semantic Occupancy Annotation via Vision-Language Guided Gaussian Splattingcited by 14
  13. 2024Segment Any 3D Object with Languagecited by 13
  14. 2024Open-vocabulary Mobile Manipulation in Unseen Dynamic Environments with 3D Semantic Mapscited by 13
  15. 2024LLM-enhanced Scene Graph Learning for Household Rearrangementcited by 12
  16. 2024Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Modelscited by 12
  17. 2025OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Mergingcited by 12
  18. 2025InstaScene: Towards Complete 3D Instance Decomposition and Reconstruction from Cluttered Scenescited by 11
  19. 2025Training-Free Hierarchical Scene Understanding for Gaussian Splatting with Superpoint Graphscited by 10
  20. 2025Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understandingcited by 10
  21. 2025Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perceptioncited by 9
  22. 2025FindAnything: Open-Vocabulary and Object-Centric Mapping for Robot Exploration in Any Environmentcited by 8
  23. 2025Details Matter for Indoor Open-vocabulary 3D Instance Segmentationcited by 8
  24. 2025OpenM3D: Open Vocabulary Multi-view Indoor 3D Object Detection without Human Annotationscited by 7
  25. 2024Open-Vocabulary SAM3D: Towards Training-free Open-Vocabulary 3D Scene Understandingcited by 6
  26. 2024Vocabulary-Free 3D Instance Segmentation with Vision and Language Assistantcited by 5
  27. 2024OpenScan: A Benchmark for Generalized Open-Vocabulary 3D Scene Understandingcited by 5
  28. 2025Leverage Cross-Attention for End-to-End Open-Vocabulary Panoptic Reconstructioncited by 5
  29. 2025OVSeg3R: Learn Open-vocabulary Instance Segmentation from 2D via 3D Reconstructioncited by 4
  30. 2024Zero-Shot Dual-Path Integration Framework for Open-Vocabulary 3D Instance Segmentationcited by 3
  31. 2024OE3DIS: Open-Ended 3D Point Cloud Instance Segmentationcited by 3
  32. 2024SA3DIP: Segment Any 3D Instance with Potential 3D Priorscited by 3
  33. 2025ShelfGaussian: Shelf-Supervised Open-Vocabulary Gaussian-based 3D Scene Understandingcited by 3
  34. 2025OpenTrack3D: Towards Accurate and Generalizable Open-Vocabulary 3D Instance Segmentationcited by 3
  35. 2025MoonSeg3R: Monocular Online Zero-Shot Segment Anything in 3D with Reconstructive Foundation Priorscited by 3
  36. 2024OpenObject-NAV: Open-Vocabulary Object-Oriented Navigation Based on Dynamic Carrier-Relationship Scene Graphcited by 2
  37. 2025GraphSeg: Segmented 3D Representations via Graph Edge Addition and Contractioncited by 2
  38. 2025Egocentric Action-aware Inertial Localization in Point Clouds with Vision-Language Guidancecited by 2

50 most cited of 81 All 81 (JSON)