Zero-Shot 3D Visual Grounding from Vision-Language Models
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
3D Visual Grounding (3DVG) seeks to locate target objects in 3D scenes using natural language descriptions, enabling downstream applications such as augmented reality and robotics. Existing approaches typically rely on labeled 3D data and predefined categories, limiting scalability to open-world settings. We present SeeGround, a zero-shot 3DVG framework that leverages 2D Vision-Language Models (VLMs) to bypass the need for 3D-specific training. To bridge the modality gap, we introduce a hybrid input format that pairs query-aligned rendered views with spatially enriched textual descriptions. Our framework incorporates two core components: a Perspective Adaptation Module that dynamically selects optimal viewpoints based on the query, and a Fusion Alignment Module that integrates visual and spatial signals to enhance localization precision. Extensive evaluations on ScanRefer and Nr3D confirm that SeeGround achieves substantial improvements over existing zero-shot baselines -- outperforming them by 7.7% and 7.1%, respectively -- and even rivals fully supervised alternatives, demonstrating strong generalization under challenging conditions.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 78 references that are in the map. First posted on arXiv 28 May 2025.
Builds on
66 of 78 references.
Closest ancestors · 1
Same direction, newest first
Other references · 65
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2023GPT-4 Technical Reportcited by 18,396
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
- 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
- 2023LERF: Language Embedded Radiance Fieldscited by 543
- 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
- 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
- 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
- 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
- 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
38 more
- 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
- 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 181
- 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
- 2021Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Trackingcited by 166
- 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 141
- 2022Multi-View Transformer for 3D Visual Groundingcited by 140
- 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
- 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
- 2023Robo3D: Towards Robust and Reliable 3D Perception against Corruptionscited by 129
- 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
- 2023Rethinking Range View Representation for LiDAR Segmentationcited by 128
- 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 125
- 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
- 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 99
- 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 94
- 2022LaserMix for Semi-Supervised LiDAR Semantic Segmentationcited by 93
- 2023SAI3D: Segment Any Instance in 3D Scenescited by 90
- 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
- 2023OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Datacited by 81
- 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
- 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
- 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 67
- 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
- 2021EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentationcited by 59
- 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
- 2024Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Drivingcited by 54
- 2023FRNet: Frustum-Range Networks for Scalable LiDAR Segmentationcited by 50
- 2022Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Groundingcited by 41
- 2024DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenescited by 38
- 2024PhysFlow: Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulationcited by 36
- 20244D Contrastive Superflows are Dense 3D Representation Learnerscited by 33
- 2023Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Groundingcited by 32
- 2024MiKASA: Multi-Key-Anchor & Scene-Aware Transformer for 3D Visual Groundingcited by 29
- 2024Agent3D-Zero: An Agent for Zero-shot 3D Understandingcited by 29
- 2023Interactive Planning Using Large Language Models for Partially Observable Robotics Taskscited by 27
- 2024Is Your LiDAR Placement Optimized for 3D Scene Understanding?cited by 26
- 2024Multi-Space Alignments Towards Universal LiDAR Segmentationcited by 25
- 2024Multi-branch Collaborative Learning Network for 3D Visual Groundingcited by 20
50 most cited of 65 All 65 (JSON)
Built on by
Citing papers per year: 2025: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding
- 2026Z3D: Zero-Shot 3D Visual Grounding from Images
- 2026Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Grounding
- 2025View-on-Graph: Zero-shot 3D Visual Grounding via Vision-Language Reasoning on Scene Graphs
- 2026SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matching
- 2026UniGround: Universal 3D Visual Grounding via Training-Free Scene Parsing
- 2026Scaling Diverse Language Generation for 3D Visual Grounding
- 2024Solving Zero-Shot 3D Visual Grounding as Constraint Satisfaction Problems