# Details Matter for Indoor Open-vocabulary 3D Instance Segmentation arXiv: https://arxiv.org/abs/2507.23134 ยท HTML: https://mlsanity.com/paper/2507.23134/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 64 of its 70 references that are in the map. - First posted on arXiv 30 Jul 2025. ## Placed in - Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/) ## Builds on (most cited first, 30 of 64) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017) - [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019) - [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024) - [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021) - [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023) - [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022) - [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021) - [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022) - [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022) - [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022) - [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023) - [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023) - [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022) - [YOLO-World: Real-Time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2401.17270/) (2024) - [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021) - [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020) - [Dynamic Filter Networks](https://mlsanity.com/paper/1605.09673/) (2016) - [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020) - [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023) - [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022) - [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023) - [Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/1906.00817/) (2019) - [Open-Vocabulary DETR with Conditional Matching](https://mlsanity.com/paper/2203.11876/) (2022) - [3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans](https://mlsanity.com/paper/1812.07003/) (2018) - [RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening](https://mlsanity.com/paper/2103.15597/) (2021) - [SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2203.01509/) (2022) ## Built on by (most cited first, 8 of 8) - [3AM: 3egment Anything with Geometric Consistency in Videos](https://mlsanity.com/paper/2601.08831/) (2026) - [OpenTrack3D: Towards Accurate and Generalizable Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2512.03532/) (2025) - [Retrieving Objects from 3D Scenes with Box-Guided Open-Vocabulary Instance Segmentation](https://mlsanity.com/paper/2512.19088/) (2025) - [MV3DIS: Multi-View Mask Matching via 3D Guides for Zero-Shot 3D Instance Segmentation](https://mlsanity.com/paper/2604.08916/) (2026) - [EvObj: Learning Evolving Object-centric Representations for 3D Instance Segmentation without Scene Supervision](https://mlsanity.com/paper/2605.13152/) (2026) - [Open-vocabulary 3D scene perception in industrial environments](https://mlsanity.com/paper/2602.19823/) (2026) - [Cross-Attentive Multiview Fusion of Vision-Language Embeddings](https://mlsanity.com/paper/2604.12551/) (2026) - [FoundObj: Self-supervised Foundation Models as Rewards for Label-free 3D Object Segmentation](https://mlsanity.com/paper/2605.27178/) (2026) ## Abstract (arXiv) Unlike closed-vocabulary 3D instance segmentation that is often trained end-to-end, open-vocabulary 3D instance segmentation (OV-3DIS) often leverages vision-language models (VLMs) to generate 3D instance proposals and classify them. While various concepts have been proposed from existing research, we observe that these individual concepts are not mutually exclusive but complementary. In this paper, we propose a new state-of-the-art solution for OV-3DIS by carefully designing a recipe to combine the concepts together and refining them to address key challenges. Our solution follows the two-stage scheme: 3D proposal generation and instance classification. We employ robust 3D tracking-based proposal aggregation to generate 3D proposals and remove overlapped or partial proposals by iterative merging/removal. For the classification stage, we replace the standard CLIP model with Alpha-CLIP, which incorporates object masks as an alpha channel to reduce background noise and obtain object-centric representation. Additionally, we introduce the standardized maximum similarity (SMS) score to normalize text-to-proposal similarity, effectively filtering out false positives and boosting precision. Our framework achieves state-of-the-art performance on ScanNet200 and S3DIS across all AP and AR metrics, even surpassing an end-to-end closed-vocabulary method.