mlsanityData to 7 Oct 2026

AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models

arXiv 2605.25901 ↗First posted 25 May 2026cs.CV cs.ROcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

3D Visual Grounding (3DVG) is an essential capability for embodied AI, requiring agents to localize objects in 3D scenes based on natural language descriptions. Recent zero-shot methods leverage 2D vision-language models (LVLMs). However, they often rely on existing sets of multi-view images and struggle with the limited semantic and spatial details provided by standard 3D segmentation tools. We present AgentGrounder, a zero-shot 3D visual grounding framework that operates directly on colored point clouds without task-specific 3D training. Our approach follows a two-stage design: (1) an offline stage that applies 3D model to build an Object Lookup Table (OLT) with instance IDs, semantic labels, 3D bounding boxes; and (2) an online tool-driven agent that decomposes each query, retrieves only relevant candidates from the OLT, performs geometric scoring, and triggers image rendering on demand when additional visual evidence (e.g., color, material, or viewpoint-sensitive cues) is required. Compared with fixed anchor-target matching pipelines, this design reduces cascading matching errors and improves context-window efficiency by avoiding prompts overloaded with irrelevant objects. We evaluate on ScanRefer and Nr3D under a zero-shot setting and observe consistent improvements over SeeGround in our setup, including +2.5% [email protected] on ScanRefer and +6.3% on Nr3D, with a notable +6.3% gain on Nr3D view-independent queries. These results show that combining selective retrieval, geometric reasoning, and adaptive visual inspection yields a practical and robust foundation for open-vocabulary 3D grounding. Our code is available at https://github.com/be2rlab/AgentGrounder.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 31 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

28 of 31 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025IRef-VLA: A Benchmark for Interactive Referential Grounding with Imperfect Language in 3D Scenes20 Mar
  2. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models2 Jan
  3. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  4. 2024Zero-shot Object Navigation with Vision-Language Models Reasoning24 Oct
  5. 2024PhyGrasp: Generalizing Robotic Grasping with Physics-informed Large Multimodal Models26 Feb

Other references · 23

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  3. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  4. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  5. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  6. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  7. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  8. 2023SAM3D: Segment Anything in 3D Scenescited by 155
  9. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  10. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  11. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  12. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
11 more
  1. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  2. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  3. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  4. 2023ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolutioncited by 52
  5. 2023Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Groundingcited by 32
  6. 2024MiKASA: Multi-Key-Anchor & Scene-Aware Transformer for 3D Visual Groundingcited by 29
  7. 2024Multi-branch Collaborative Learning Network for 3D Visual Groundingcited by 20
  8. 2024Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Trackingcited by 14
  9. 2024Transcrib3D: 3D Referring Expression Resolution through Large Language Modelscited by 10
  10. 2025SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Modelscited by 10
  11. 2024Solving Zero-Shot 3D Visual Grounding as Constraint Satisfaction Problemscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Grounding
  2. 2026SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matching
  3. 2025Zero-Shot 3D Visual Grounding from Vision-Language Models
  4. 2025OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding
  5. 2026UniGround: Universal 3D Visual Grounding via Training-Free Scene Parsing
  6. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding
  7. 2025View-on-Graph: Zero-shot 3D Visual Grounding via Vision-Language Reasoning on Scene Graphs
  8. 2026Z3D: Zero-Shot 3D Visual Grounding from Images