mlsanityData to 14 Oct 2026

LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent

arXiv 2309.12311 ↗First posted 21 Sep 2023cs.CV cs.AI cs.CLcited by 135

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

3D visual grounding is a critical skill for household robots, enabling them to navigate, manipulate objects, and answer questions based on their environment. While existing approaches often rely on extensive labeled data or exhibit limitations in handling complex language queries, we propose LLM-Grounder, a novel zero-shot, open-vocabulary, Large Language Model (LLM)-based 3D visual grounding pipeline. LLM-Grounder utilizes an LLM to decompose complex natural language queries into semantic constituents and employs a visual grounding tool, such as OpenScene or LERF, to identify objects in a 3D scene. The LLM then evaluates the spatial and commonsense relations among the proposed objects to make a final grounding decision. Our method does not require any labeled training data and can generalize to novel 3D scenes and arbitrary text queries. We evaluate LLM-Grounder on the ScanRefer benchmark and demonstrate state-of-the-art zero-shot grounding accuracy. Our findings indicate that LLMs significantly improve the grounding capability, especially for complex language queries, making LLM-Grounder an effective approach for 3D vision-language tasks in robotics. Videos and interactive demos can be found on the project website https://chat-with-nerf.github.io/ .

Key facts. Cited by 135 papers in the map (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 49 of its 62 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

49 of 62 references.

References · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  9. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
37 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  2. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  5. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  7. 2022Language-driven Semantic Segmentationcited by 717
  8. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  9. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  10. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  11. 2023LERF: Language Embedded Radiance Fieldscited by 543
  12. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  13. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  14. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  15. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  16. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  17. 2022Visual Language Maps for Robot Navigationcited by 365
  18. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  19. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  20. 2022TALM: Tool Augmented Language Modelscited by 197
  21. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  22. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  23. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  24. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  25. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 156
  26. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 148
  27. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 141
  28. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  29. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  30. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  31. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 89
  32. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  33. 2023NS3D: Neuro-Symbolic Grounding of 3D Objects and Relationscited by 57
  34. 2022Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Groundingcited by 41
  35. 2022Feature-Realistic Neural Fusion for Real-Time, Open Set Scene Understandingcited by 37
  36. 2021Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Imagescited by 33
  37. 2022Learning Point-Language Hierarchical Alignment for 3D Visual Groundingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 52; 2025: 46; 2026: 30.

135 papers.

Sort:

By area:

  1. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  2. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 208
  3. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  4. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  5. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  6. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  7. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 75
  8. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 67
  9. 2024KnowAgent: Knowledge-Augmented Planning for LLM-Based Agentscited by 61
  10. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  11. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  12. 2024Multi-modal Situated Reasoning in 3D Scenescited by 51
38 more
  1. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 51
  2. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  3. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  4. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  5. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  6. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 39
  7. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  8. 2024Situational Awareness Matters in 3D Vision Language Reasoningcited by 33
  9. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understandingcited by 32
  10. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  11. 2024DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulationcited by 32
  12. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  13. 2023Think, Act, and Ask: Open-World Interactive Personalized Robot Navigationcited by 31
  14. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  15. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 27
  16. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 27
  17. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 25
  18. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 23
  19. 2025Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3Dcited by 23
  20. 2025Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactionscited by 19
  21. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentationcited by 16
  22. 2024ChatCam: Empowering Camera Control through Conversational AIcited by 15
  23. 2024EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editingcited by 15
  24. 2023Advances in Embodied Navigation Using Large Language Models: A Surveycited by 14
  25. 2024VURF: A General-purpose Reasoning and Self-refinement Framework for Video Understandingcited by 13
  26. 2024SceneGenAgent: Precise Industrial Scene Generation with Coding Agentcited by 13
  27. 2024Idea23D: Collaborative LMM Agents Enable 3D Model Generation from Interleaved Multimodal Inputscited by 12
  28. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoningcited by 12
  29. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Groundingcited by 12
  30. 20253EED: Ground Everything Everywhere in 3Dcited by 12
  31. 2023LgTS: Dynamic Task Sampling using LLM-generated sub-goals for Reinforcement Learning Agentscited by 11
  32. 2024Transcrib3D: 3D Referring Expression Resolution through Large Language Modelscited by 11
  33. 2024Grounding 3D Scene Affordance From Egocentric Interactionscited by 11
  34. 2025SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Modelscited by 11
  35. 2024Semantics-Controlled Gaussian Splatting for Outdoor Scene Reconstruction and Rendering in Virtual Realitycited by 10
  36. 2024From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impactscited by 10
  37. 2024PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal Modelcited by 9
  38. 2024Task-oriented Sequential Grounding and Navigation in 3D Scenescited by 9

50 most cited of 135 All 135 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Relational Scene Graphs for Object Grounding of Natural Language Commands
  2. 2026Scaling Diverse Language Generation for 3D Visual Grounding
  3. 2025A Neural Representation Framework with LLM-Driven Spatial Reasoning for Open-Vocabulary 3D Visual Grounding
  4. 2025Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning
  5. 20233D-LLM: Injecting the 3D World into Large Language Models
  6. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
  7. 2026GROW2: Grounding Which and Where for Robot Tool Use
  8. 2026Ground3D-LMM: Fine-Grained 3D Point Grounding and Spatial Reasoning with LMM