mlsanityData to 14 Oct 2026

VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding

arXiv 2410.13860 ↗First posted 17 Oct 2024cs.CV cs.ROcited by 56

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

3D visual grounding is crucial for robots, requiring integration of natural language and 3D scene understanding. Traditional methods depending on supervised learning with 3D point clouds are limited by scarce datasets. Recently zero-shot methods leveraging LLMs have been proposed to address the data issue. While effective, these methods only use object-centric information, limiting their ability to handle complex queries. In this work, we present VLM-Grounder, a novel framework using vision-language models (VLMs) for zero-shot 3D visual grounding based solely on 2D images. VLM-Grounder dynamically stitches image sequences, employs a grounding and feedback scheme to find the target object, and uses a multi-view ensemble projection to accurately estimate 3D bounding boxes. Experiments on ScanRefer and Nr3D datasets show VLM-Grounder outperforms previous zero-shot methods, achieving 51.6% [email protected] on ScanRefer and 48.0% Acc on Nr3D, without relying on 3D geometry or object priors. Codes are available at https://github.com/OpenRobotLab/VLM-Grounder .

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Builds on 44 of its 56 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

44 of 56 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Segment Anythingcited by 8,234
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
32 more
  1. 2016ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Camerascited by 806
  2. 2018Open3D: A Modern Library for 3D Data Processingcited by 721
  3. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  4. 2023LERF: Language Embedded Radiance Fieldscited by 543
  5. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  6. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  7. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  8. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  9. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  10. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
  11. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  12. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  13. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  14. 2023NICER-SLAM: Neural Implicit Scene Encoding for RGB SLAMcited by 164
  15. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 147
  16. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 141
  17. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  18. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  19. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  20. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  21. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  22. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 122
  23. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  24. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 89
  25. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  26. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  27. 2023Dense RGB SLAM with Neural Implicit Mapscited by 42
  28. 2023PATS: Patch Area Transportation with Subdivision for Local Feature Matchingcited by 33
  29. 2023Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphscited by 33
  30. 2023Multi-Modal Neural Radiance Field for Monocular Dense SLAM with a Light-Weight ToF Sensorcited by 30
  31. 2024Agent3D-Zero: An Agent for Zero-shot 3D Understandingcited by 29
  32. 2024TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answeringcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 27; 2026: 25.

56 papers.

Sort:

By area:

  1. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 183
  2. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 138
  3. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  4. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 107
  5. 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialoguescited by 85
  6. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 67
  7. 2025Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Modelscited by 52
  8. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 51
  9. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  10. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 33
  11. 2025N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Modelscited by 29
  12. 2025Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3Dcited by 23
38 more
  1. 2025Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactionscited by 19
  2. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Groundingcited by 12
  3. 2025SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Modelscited by 11
  4. 2025Language-to-Space Programming for Training-Free 3D Visual Groundingcited by 8
  5. 2024Solving Zero-Shot 3D Visual Grounding as Constraint Satisfaction Problemscited by 7
  6. 2025SeqVLM: Proposal-Guided Multi-View Sequences Reasoning via VLM for Zero-Shot 3D Visual Groundingcited by 7
  7. 2026PoseVLA: Universal Pose Pretraining for Generalizable Vision-Language-Action Policiescited by 7
  8. 2025S2-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidancecited by 5
  9. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generationcited by 4
  10. 2026See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoningcited by 4
  11. 2025Warehouse Spatial Question Answering with LLM Agentcited by 3
  12. 2025Video4Spatial: Towards Visuospatial Intelligence with Context-Guided Video Generationcited by 3
  13. 2025PanoGrounder: Bridging 2D and 3D with Panoramic Scene Representations for VLM-based 3D Visual Groundingcited by 3
  14. 2025Zero-Shot 3D Visual Grounding from Vision-Language Modelscited by 2
  15. 2025FreeQ-Graph: Free-form Querying with Semantic Consistent Scene Graph for 3D Scene Understandingcited by 2
  16. 2026Z3D: Zero-Shot 3D Visual Grounding from Imagescited by 2
  17. 2026Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modelingcited by 2
  18. 2025T-Rex: Task-Adaptive Spatial Representation Extraction for Robotic Manipulation with Vision-Language Modelscited by 1
  19. 2025ChangingGrounding: 3D Visual Grounding in Changing Scenescited by 1
  20. 2026VIZOR: Viewpoint-Invariant Zero-Shot Scene Graph Generation for 3D Scene Reasoningcited by 1
  21. 2026UniGround: Universal 3D Visual Grounding via Training-Free Scene Parsingcited by 1
  22. 2026Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Groundingcited by 1
  23. 2026Perceive-then-Plan: Layout-as-Policy for Monocular 3D Scene Layout Estimationcited by 1
  24. 2026Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoningcited by 1
  25. 2026G2TAM: Geometry Grounded Track Anything Modelcited by 1
  26. 2026Seeing Once is Enough? Online Geometry-Aware Token Pruning for 3D Question Answeringcited by 1
  27. 2025DSM: Constructing a Diverse Semantic Map for 3D Visual Groundingcited by 0
  28. 2025Handle-based Mesh Deformation Guided By Vision Language Modelcited by 0
  29. 2025Disc3D: Automatic Curation of High-Quality 3D Dialog Data via Discriminative Object Referringcited by 0
  30. 2025OpenGround: Planning-based Online Perception for Open-World 3D Visual Groundingcited by 0
  31. 2026SpatialPoint: Spatial-aware Point Prediction for Embodied Localizationcited by 0
  32. 2026Multiple Consistent 2D-3D Mappings for Robust Zero-Shot 3D Visual Groundingcited by 0
  33. 2026SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matchingcited by 0
  34. 2026AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Modelscited by 0
  35. 2026SSR3D-LLM: Structured Spatial Reasoning via Latent Steps for Fine-Grained Grounding in Unified 3D-LLMscited by 0
  36. 2026ZeroDex: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoningcited by 0
  37. 2026Agentic Collaborative Cognition for Zero-Shot 3D Understandingcited by 0
  38. 2026PruneGround: Plug-and-play Spatial Pruning for 3D Visual Groundingcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025View-on-Graph: Zero-shot 3D Visual Grounding via Vision-Language Reasoning on Scene Graphs
  2. 2026GuideGround: VLM-guided Semantic Understanding and Viewpoint-aware Reasoning for 3D Visual Grounding
  3. 2023Weakly-Supervised 3D Visual Grounding based on Visual Language Alignment
  4. 2026Scaling Diverse Language Generation for 3D Visual Grounding
  5. 2026Toward Comprehensive 3D Grounding: Orientation Grounding through Vision-Language Models
  6. 2026Reasoning Matters for 3D Visual Grounding
  7. 2025ReasonGrounder: LVLM-Guided Hierarchical Feature Splatting for Open-Vocabulary 3D Visual Grounding and Reasoning
  8. 2026StrataVLA: Hierarchical and Efficient 3D Geometric Grounding for Vision-Language-Action Models