mlsanityData to 14 Oct 2026

NVSMask3D: Hard Visual Prompting with Camera Pose Interpolation for 3D Open Vocabulary Instance Segmentation

arXiv 2504.14638 ↗First posted 20 Apr 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-language models (VLMs) have demonstrated impressive zero-shot transfer capabilities in image-level visual perception tasks. However, they fall short in 3D instance-level segmentation tasks that require accurate localization and recognition of individual objects. To bridge this gap, we introduce a novel 3D Gaussian Splatting based hard visual prompting approach that leverages camera interpolation to generate diverse viewpoints around target objects without any 2D-3D optimization or fine-tuning. Our method simulates realistic 3D perspectives, effectively augmenting existing hard visual prompts by enforcing geometric consistency across viewpoints. This training-free strategy seamlessly integrates with prior hard visual prompts, enriching object-descriptive features and enabling VLMs to achieve more robust and accurate 3D instance segmentation in diverse 3D scenes.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 40 references that are in the map. First posted on arXiv 20 Apr 2025.

Builds on

39 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views4 Apr

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Segment Anythingcited by 8,234
  3. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  7. 2022Visual Prompt Tuningcited by 1,681
  8. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  9. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  10. 2022Language-driven Semantic Segmentationcited by 717
  11. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 600
  12. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
26 more
  1. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  2. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  3. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  4. 2023LangSplat: 3D Language Gaussian Splattingcited by 415
  5. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  6. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
  7. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  8. 2024gsplat: An Open-Source Library for Gaussian Splattingcited by 285
  9. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  10. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  11. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  12. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 181
  13. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  14. 2023Dense Text-to-Image Generation with Attention Modulationcited by 163
  15. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 124
  16. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 99
  17. 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 94
  18. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  19. 2023Fine-Grained Visual Promptingcited by 82
  20. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  21. 2023OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Datacited by 81
  22. 2023Improving Visual Prompt Tuning for Self-supervised Vision Transformerscited by 57
  23. 2023Efficient 3D Semantic Segmentation with Superpoint Transformercited by 57
  24. 2019Point Cloud Oversegmentation with Graph-Structured Deep Metric Learningcited by 42
  25. 2023Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understandingcited by 38
  26. 2024Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentationcited by 30

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025OpenSplat3D: Open-Vocabulary 3D Instance Segmentation using Gaussian Splatting
  2. 2025GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond
  3. 2026Open-Vocabulary and Referring Segmentation for 3D Gaussians Using 2D Detectors
  4. 2025Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation
  5. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models
  6. 2025FOLK: Fast Open-Vocabulary 3D Instance Segmentation via Label-guided Knowledge Distillation
  7. 2024GAGS: Granularity-Aware Feature Distillation for Language Gaussian Splatting
  8. 2025Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration