mlsanityData to 7 Oct 2026

OpenMask3D: Open-Vocabulary 3D Instance Segmentation

arXiv 2306.13631 ↗First posted 23 Jun 2023cs.CVcited by 288

Not on the map. Why?

Cited by 29 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce the task of open-vocabulary 3D instance segmentation. Current approaches for 3D instance segmentation can typically only recognize object categories from a pre-defined closed set of classes that are annotated in the training datasets. This results in important limitations for real-world applications where one might need to perform tasks guided by novel, open-vocabulary queries related to a wide variety of objects. Recently, open-vocabulary 3D scene understanding methods have emerged to address this problem by learning queryable features for each point in the scene. While such a representation can be directly employed to perform semantic segmentation, existing methods cannot separate multiple object instances. In this work, we address this limitation, and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D instance segmentation. Guided by predicted class-agnostic 3D instance masks, our model aggregates per-mask features via multi-view fusion of CLIP-based image embeddings. Experiments and ablation studies on ScanNet200 and Replica show that OpenMask3D outperforms other open-vocabulary methods, especially on the long-tail distribution. Qualitative experiments further showcase OpenMask3D's ability to segment object properties based on free-form queries describing geometry, affordances, and materials.

Key facts. Cited by 288 papers in the map (data through 7 Oct 2026). Builds on 64 of its 81 references that are in the map. First posted on arXiv 23 Jun 2023.

Builds on

64 of 81 references.

References · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Segment Anythingcited by 8,158
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  7. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  12. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
38 more
  1. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  2. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  4. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  5. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  6. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  7. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  8. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  9. 2022Language-driven Semantic Segmentationcited by 711
  10. 2021Extract Free Dense Labels from CLIPcited by 580
  11. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  12. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  13. 2023LERF: Language Embedded Radiance Fieldscited by 538
  14. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  15. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  16. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  17. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  18. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 384
  19. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  20. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  21. 2022Visual Language Maps for Robot Navigationcited by 362
  22. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  23. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  24. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  25. 2018SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filterscited by 261
  26. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  27. 20183D-SIS: 3D Semantic Instance Segmentation of RGB-D Scanscited by 216
  28. 2021Decoupling Zero-Shot Semantic Segmentationcited by 198
  29. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 197
  30. 2017SEGCloud: Semantic Segmentation of 3D Point Cloudscited by 178
  31. 2020OccuSeg: Occupancy-aware 3D Instance Segmentationcited by 177
  32. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  33. 2017SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentationcited by 168
  34. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  35. 2019FSS-1000: A 1000-Class Dataset for Few-Shot Segmentationcited by 160
  36. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  37. 2019Learning Object Bounding Boxes for 3D Instance Segmentation on Point Cloudscited by 156
  38. 2018Point Convolutional Neural Networks by Extension Operatorscited by 154

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 90; 2025: 90; 2026: 90.

288 papers.

Sort:

By area:

  1. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 387
  2. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  3. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205
  4. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  5. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  6. 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 141
  7. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  8. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  9. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  10. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  11. 2025PARTFIELD: Learning 3D Feature Fields for Part Segmentation and Beyondcited by 91
  12. 2023SAI3D: Segment Any Instance in 3D Scenescited by 90
38 more
  1. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  2. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  3. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 81
  4. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  5. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  6. 2023OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learningcited by 75
  7. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  8. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 70
  9. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 68
  10. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  11. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  12. 2023Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labelscited by 64
  13. 2023Fully Sparse 3D Occupancy Predictioncited by 61
  14. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  15. 2024MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentationcited by 56
  16. 2024Part123: Part-aware 3D Reconstruction from a Single-view Imagecited by 54
  17. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
  18. 2025Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registrationcited by 50
  19. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  20. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  21. 2024Point-SAM: Promptable 3D Segmentation Model for Point Cloudscited by 49
  22. 2024Better Call SAL: Towards Learning to Segment Anything in Lidarcited by 47
  23. 2024EgoLifter: Open-world 3D Segmentation for Egocentric Perceptioncited by 46
  24. 2023SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentationcited by 44
  25. 2025Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representationscited by 42
  26. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  27. 2024EmbodiedSAM: Online Segment Any 3D Thing in Real Timecited by 40
  28. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 34
  29. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  30. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  31. 2024Search3D: Hierarchical Open-Vocabulary 3D Segmentationcited by 31
  32. 2024Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentationcited by 30
  33. 2024Find Any Part in 3Dcited by 30
  34. 2024SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphscited by 29
  35. 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Modelcited by 27
  36. 2024ODIN: A Single Model for 2D and 3D Segmentationcited by 25
  37. 2025ObjectGS: Object-aware Scene Reconstruction and Scene Understanding via Gaussian Splattingcited by 24
  38. 2025Lifting by Gaussians: A Simple, Fast and Flexible Method for 3D Instance Segmentationcited by 23

50 most cited of 288 All 288 (JSON)