mlsanityData to 7 Oct 2026

Cross3DVG: Cross-Dataset 3D Visual Grounding on Different RGB-D Scans

arXiv 2305.13876 ↗First posted 23 May 2023cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

We present a novel task for cross-dataset visual grounding in 3D scenes (Cross3DVG), which overcomes limitations of existing 3D visual grounding models, specifically their restricted 3D resources and consequent tendencies of overfitting a specific 3D dataset. We created RIORefer, a large-scale 3D visual grounding dataset, to facilitate Cross3DVG. It includes more than 63k diverse descriptions of 3D objects within 1,380 indoor RGB-D scans from 3RScan, with human annotations. After training the Cross3DVG model using the source 3D visual grounding dataset, we evaluate it without target labels using the target dataset with, e.g., different sensors, 3D reconstruction methods, and language annotators. Comprehensive experiments are conducted using established visual grounding models and with CLIP-based multi-view 2D and 3D integration designed to bridge gaps among 3D datasets. For Cross3DVG tasks, (i) cross-dataset 3D visual grounding exhibits significantly worse performance than learning and evaluation with a single dataset because of the 3D data and language variants across datasets. Moreover, (ii) better object detector and localization modules and fusing 3D data and multi-view CLIP-based image features can alleviate this lower performance. Our Cross3DVG task can provide a benchmark for developing robust 3D visual grounding models to handle diverse 3D scenes while leveraging deep language understanding.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 47 of its 65 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

47 of 65 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  6. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  7. 2017Decoupled Weight Decay Regularizationcited by 1,722
  8. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  9. 2016Modeling Context in Referring Expressionscited by 1,054
  10. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  11. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  12. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
35 more
  1. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  2. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  3. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  4. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  5. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  6. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  7. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  8. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  9. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  10. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  11. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 197
  12. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 187
  13. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  14. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  15. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  16. 2021Hierarchical Aggregation for 3D Instance Segmentationcited by 126
  17. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  18. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  19. 2020ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votescited by 119
  20. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  21. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  22. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 98
  23. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88
  24. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  25. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 77
  26. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  27. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  28. 2019Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Videocited by 60
  29. 2022Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Groundingcited by 41
  30. 2022Spatiality-guided Transformer for 3D Dense Captioning on Point Cloudscited by 40
  31. 2021Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Imagescited by 33
  32. 2022ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenescited by 32
  33. 2016Zero-Shot Visual Question Answeringcited by 20
  34. 2022Toward Explainable and Fine-Grained 3D Grounding through Referring Textual Phrasescited by 16
  35. 2021Domain-robust VQA with diverse datasets and methods but no target labelscited by 11

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 5.

10 papers.

  1. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  2. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 26
  3. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 24
  4. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Groundingcited by 12
  5. 2024BIP3D: Bridging 2D Images and 3D Perception for Embodied Intelligencecited by 10
  6. 2025PanoGrounder: Bridging 2D and 3D with Panoramic Scene Representations for VLM-based 3D Visual Groundingcited by 3
  7. 2025RGBT-GroundBench: Visual Grounding Beyond RGB in Complex Real-World Scenarioscited by 3
  8. 2025TriCLIP-3D: A Unified Parameter-Efficient Framework for Tri-Modal 3D Visual Grounding based on CLIPcited by 2
  9. 2025Propagating Sparse Depth via Depth Foundation Model for Out-of-Distribution Depth Completioncited by 2
  10. 2024RegTrack: Simplicity Beneath Complexity in Robust Multi-Modal 3D Multi-Object Trackingcited by 0