mlsanityData to 14 Oct 2026

Scene-Q: Confidence-Aware Coarse-to-Fine Querying of 3D Scenes with Selective VLM Reasoning

arXiv 2609.20235 ↗First posted 28 Jul 2026cs.CV cs.ROcited by 0

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Indoor mobile robots require open-vocabulary scene understanding that grounds natural-language queries in a consistent 3D map. Many existing systems ultimately rely on cosine-similarity retrieval with contrastive image--text encoders, which is efficient but brittle when labels are near-synonymous or multiple similar instances appear. We present Scene-Q, a confidence-aware coarse-to-fine querying framework that normalizes encoder scores with temperature scaling and selectively invokes a reasoning VLM only for low-confidence cases. High-confidence queries are answered by fast retrieval, while ambiguous ones are reranked over a small top-K candidate set using the original multi-view images and instance bounding boxes, enabling context-aware disambiguation at low cost. Scene-Q improves open-vocabulary 3D instance segmentation on ScanNet200 and natural-language 3D instance retrieval on real-world reconstructions, with the largest gains on spatial and relational queries while keeping a substantial fraction of queries on the fast path.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 43 references that are in the map. First posted on arXiv 28 Jul 2026.

Builds on

42 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Segment Anythingcited by 8,234
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
29 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  3. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  4. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  5. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  7. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  8. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  9. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  10. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  11. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  12. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  13. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
  14. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  15. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  16. 2023SAM3D: Segment Anything in 3D Scenescited by 157
  17. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 124
  18. 2023OneFormer3D: One Transformer for Unified Point Cloud Segmentationcited by 95
  19. 2022High-Quality Entity Segmentationcited by 93
  20. 2023SAI3D: Segment Any Instance in 3D Scenescited by 90
  21. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  22. 2023OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Datacited by 81
  23. 2024RegionGPT: Towards Region Understanding Vision Language Modelcited by 71
  24. 2024RTAB-Map as an Open-Source Lidar and Visual SLAM Library for Large-Scale and Long-Term Online Operationcited by 62
  25. 2024MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentationcited by 56
  26. 2023ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolutioncited by 53
  27. 2023SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentationcited by 44
  28. 2024Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentationcited by 30
  29. 2025OV-MAP: Open-Vocabulary Zero-Shot 3D Instance Segmentation Map for Robotscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Vision-Language-Motion Maps: An Open-Vocabulary, Uncertainty-Aware, Queryable Motion Attribute for 3D Scene Maps ↗
  2. 2025Queryable 3D Scene Representation: A Multi-Modal Framework for Semantic Reasoning and Robotic Task Planning
  3. 2024VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation
  4. 2025Open-World 3D Scene Graph Generation for Retrieval-Augmented Reasoning
  5. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graph
  6. 2026OVIP-SG: Open-Vocabulary Instance-Preserving Scene Graphs for Mapping and Retrieval of Small, Fine-Grained Objects ↗
  7. 2026INHerit-SG: Incremental Hierarchical Semantic Scene Graphs for Complex Embodied Retrieval
  8. 2026SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes