mlsanityData to 14 Oct 2026

Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding

arXiv 2311.15383 ↗First posted 26 Nov 2023cs.CVcited by 64

Not on the map. Why?

Abstract (arXiv)

3D Visual Grounding (3DVG) aims at localizing 3D object based on textual descriptions. Conventional supervised methods for 3DVG often necessitate extensive annotations and a predefined vocabulary, which can be restrictive. To address this issue, we propose a novel visual programming approach for zero-shot open-vocabulary 3DVG, leveraging the capabilities of large language models (LLMs). Our approach begins with a unique dialog-based method, engaging with LLMs to establish a foundational understanding of zero-shot 3DVG. Building on this, we design a visual program that consists of three types of modules, i.e., view-independent, view-dependent, and functional modules. These modules, specifically tailored for 3D scenarios, work collaboratively to perform complex reasoning and inference. Furthermore, we develop an innovative language-object correlation module to extend the scope of existing 3D object detectors into open-vocabulary scenarios. Extensive experiments demonstrate that our zero-shot approach can outperform some supervised baselines, marking a significant stride towards effective 3DVG.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Builds on 58 of its 74 references that are in the map. First posted on arXiv 26 Nov 2023.

Builds on

58 of 74 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  10. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  12. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
38 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  3. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  5. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  6. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  7. 2022Language-driven Semantic Segmentationcited by 717
  8. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  9. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  10. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  11. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  12. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  13. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  14. 2023LERF: Language Embedded Radiance Fieldscited by 543
  15. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  16. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  17. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  18. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  19. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  20. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  21. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 436
  22. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 321
  23. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  24. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
  25. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  26. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  27. 2021Group-Free 3D Object Detection via Transformerscited by 215
  28. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 200
  29. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 188
  30. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 187
  31. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 141
  32. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  33. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  34. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  35. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 122
  36. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 99
  37. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 89
  38. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 33; 2026: 16.

64 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  2. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 67
  3. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 63
  4. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  5. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 51
  6. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  7. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  8. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  9. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 27
  10. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 25
  11. 20243DSceneEditor: Controllable 3D Scene Editing with Gaussian Splattingcited by 22
  12. 2025URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Modelcited by 21
38 more
  1. 2024Space3D-Bench: Spatial 3D Question Answering Benchmarkcited by 17
  2. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Groundingcited by 12
  3. 20253EED: Ground Everything Everywhere in 3Dcited by 12
  4. 2025SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Modelscited by 11
  5. 2025A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Modelscited by 10
  6. 2025AerialVG: A Challenging Benchmark for Aerial Visual Grounding by Exploring Positional Relationscited by 10
  7. 2024Task-oriented Sequential Grounding and Navigation in 3D Scenescited by 9
  8. 2025Language-to-Space Programming for Training-Free 3D Visual Groundingcited by 8
  9. 2025ViewSRD: 3D Visual Grounding via Structured Multi-View Decompositioncited by 8
  10. 2025Towards Scalable Spatial Intelligence via 2D-to-3D Data Liftingcited by 8
  11. 2024Solving Zero-Shot 3D Visual Grounding as Constraint Satisfaction Problemscited by 7
  12. 2025ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Groundingcited by 7
  13. 2025Empowering Large Language Models with 3D Situation Awarenesscited by 6
  14. 2025Talk2Event: Grounded Understanding of Dynamic Scenes from Event Camerascited by 6
  15. 2024Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attentioncited by 4
  16. 2025From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenescited by 4
  17. 2025GroundFlow: A Plug-in Module for Temporal Reasoning on 3D Point Cloud Sequential Groundingcited by 4
  18. 2024Instance-free Text to Point Cloud Localization with Relative Position Awarenesscited by 3
  19. 2025Warehouse Spatial Question Answering with LLM Agentcited by 3
  20. 2025Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptionscited by 3
  21. 2025InteractMove: Text-Controlled Human-Object Interaction Generation in 3D Scenes with Movable Objectscited by 3
  22. 2025PanoGrounder: Bridging 2D and 3D with Panoramic Scene Representations for VLM-based 3D Visual Groundingcited by 3
  23. 2024Automatic benchmarking of large multimodal models via iterative experiment programmingcited by 2
  24. 2025Zero-Shot 3D Visual Grounding from Vision-Language Modelscited by 2
  25. 2025FreeQ-Graph: Free-form Querying with Semantic Consistent Scene Graph for 3D Scene Understandingcited by 2
  26. 2025Open-Vocabulary Indoor Object Grounding with 3D Hierarchical Scene Graphcited by 2
  27. 2025TriCLIP-3D: A Unified Parameter-Efficient Framework for Tri-Modal 3D Visual Grounding based on CLIPcited by 2
  28. 2025Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models?cited by 2
  29. 2026Z3D: Zero-Shot 3D Visual Grounding from Imagescited by 2
  30. 2025Visual Grounding from Event Camerascited by 1
  31. 2025LLM-RG: Referential Grounding in Outdoor Scenarios using Large Language Modelscited by 1
  32. 2025ChangingGrounding: 3D Visual Grounding in Changing Scenescited by 1
  33. 2025Error-Driven Scene Editing for 3D Grounding in Large Language Modelscited by 1
  34. 2026VIZOR: Viewpoint-Invariant Zero-Shot Scene Graph Generation for 3D Scene Reasoningcited by 1
  35. 2026UniGround: Universal 3D Visual Grounding via Training-Free Scene Parsingcited by 1
  36. 2026Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Groundingcited by 1
  37. 2026THOM: Generating Physically Plausible Hand-Object Meshes From Textcited by 1
  38. 2025DSM: Constructing a Diverse Semantic Map for 3D Visual Groundingcited by 0

50 most cited of 64 All 64 (JSON)