mlsanityData to 7 Oct 2026

Weakly-Supervised 3D Visual Grounding based on Visual Language Alignment

arXiv 2312.09625 ↗First posted 15 Dec 2023cs.CV cs.CLcited by 3

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Learning to ground natural language queries to target objects or regions in 3D point clouds is quite essential for 3D scene understanding. Nevertheless, existing 3D visual grounding approaches require a substantial number of bounding box annotations for text queries, which is time-consuming and labor-intensive to obtain. In this paper, we propose 3D-VLA, a weakly supervised approach for 3D visual grounding based on Visual Linguistic Alignment. Our 3D-VLA exploits the superior ability of current large-scale vision-language models (VLMs) on aligning the semantics between texts and 2D images, as well as the naturally existing correspondences between 2D images and 3D point clouds, and thus implicitly constructs correspondences between texts and 3D point clouds with no need for fine-grained box annotations in the training procedure. During the inference stage, the learned text-3D correspondence will help us ground the text queries to the 3D target objects even without 2D images. To the best of our knowledge, this is the first work to investigate 3D visual grounding in a weakly supervised manner by involving large scale vision-language models, and extensive experiments on ReferIt3D and ScanRefer datasets demonstrate that our 3D-VLA achieves comparable and even superior results over the fully supervised methods.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 75 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

55 of 75 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  8. 2020Supervised Contrastive Learningcited by 2,803
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  12. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
38 more
  1. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  2. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  3. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  4. 2023LERF: Language Embedded Radiance Fieldscited by 538
  5. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  6. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  7. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  8. 2019Attention on Attention for Image Captioningcited by 289
  9. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  10. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  11. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  12. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  13. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  14. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  15. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  16. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  17. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  18. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  19. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  20. 2015Grounding of Textual Phrases in Images by Reconstructioncited by 112
  21. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  22. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  23. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 98
  24. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88
  25. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  26. 2017Grounding Referring Expressions in Images by Variational Contextcited by 82
  27. 2021Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloudcited by 81
  28. 2022Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoningcited by 69
  29. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  30. 2019Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignmentcited by 51
  31. 2020Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillationcited by 51
  32. 2022Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Groundingcited by 41
  33. 2019Adaptive Reconstruction Network for Weakly Supervised Referring Expression Groundingcited by 37
  34. 2021Relation-aware Instance Refinement for Weakly Supervised Visual Groundingcited by 35
  35. 2023CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Groundingcited by 35
  36. 2018Knowledge Aided Consistency for Weakly Supervised Phrase Groundingcited by 32
  37. 2023Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Groundingcited by 32
  38. 2021CityFlow-NL: Tracking and Retrieval of Vehicles at City Scale by Natural Language Descriptionscited by 26

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  2. 2025Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Modelcited by 2
  3. 2025DBGroup: Dual-Branch Point Grouping for Weakly Supervised 3D Semantic Instance Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Diverse Language Generation for 3D Visual Grounding
  2. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding
  3. 2023Advancing Visual Grounding with Scene Knowledge: Benchmark and Method
  4. 2026Reasoning Matters for 3D Visual Grounding
  5. 2026GuideGround: VLM-guided Semantic Understanding and Viewpoint-aware Reasoning for 3D Visual Grounding
  6. 2025From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs
  7. 2025Zero-Shot 3D Visual Grounding from Vision-Language Models
  8. 2026TDVR: Joint Text Disambiguation and Viewpoint Reasoning for Zero-Shot 3D Visual Grounding