mlsanityData to 14 Oct 2026

WildRefer: 3D Object Localization in Large-scale Dynamic Scenes with Multi-modal Visual Data and Natural Language

arXiv 2304.05645 ↗First posted 12 Apr 2023cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

We introduce the task of 3D visual grounding in large-scale dynamic scenes based on natural linguistic descriptions and online captured multi-modal visual data, including 2D images and 3D LiDAR point clouds. We present a novel method, dubbed WildRefer, for this task by fully utilizing the rich appearance information in images, the position and geometric clues in point cloud as well as the semantic knowledge of language descriptions. Besides, we propose two novel datasets, i.e., STRefer and LifeRefer, which focus on large-scale human-centric daily-life scenarios accompanied with abundant 3D object and natural language annotations. Our datasets are significant for the research of 3D visual grounding in the wild and has huge potential to boost the development of autonomous driving and service robots. Extensive experiments and ablation studies demonstrate that our method achieves state-of-the-art performance on the proposed benchmarks. The code is provided in https://github.com/4DVLab/WildRefer.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 51 of its 61 references that are in the map. First posted on arXiv 12 Apr 2023.

Builds on

51 of 61 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  9. 2016Modeling Context in Referring Expressionscited by 1,063
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  12. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
38 more
  1. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  2. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 466
  3. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  4. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  5. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  6. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  7. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 336
  8. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 271
  9. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 251
  10. 2022DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detectioncited by 224
  11. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 195
  12. 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
  13. 2019MVX-Net: Multimodal VoxelNet for 3D Object Detectioncited by 163
  14. 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 158
  15. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 141
  16. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  17. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 139
  18. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 138
  19. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  20. 2020ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votescited by 119
  21. 2016GuessWhat?! Visual object discovery through multi-modal dialoguecited by 113
  22. 2019A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehensioncited by 111
  23. 2019Dynamic Graph Attention for Referring Expression Comprehensioncited by 111
  24. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 110
  25. 2016Modeling Relationships in Referential Expressions with Compositional Modular Networkscited by 106
  26. 2018Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networkscited by 102
  27. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 99
  28. 2021Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloudcited by 81
  29. 2018Grounded Video Descriptioncited by 78
  30. 2019CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressionscited by 73
  31. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  32. 2021EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentationcited by 59
  33. 2021Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR-based Perceptioncited by 57
  34. 2022Text2Pos: Text-to-Point-Cloud Cross-Modal Localizationcited by 40
  35. 2022STCrowd: A Multimodal Dataset for Pedestrian Perception in Crowded Scenescited by 35
  36. 2021Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Imagescited by 33
  37. 2022Weakly Supervised 3D Multi-person Pose Estimation for Large-scale Scenes based on Monocular Camera and Single LiDARcited by 23
  38. 2021Deconfounded Visual Groundingcited by 16

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 5.

8 papers.

  1. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 25
  2. 20253EED: Ground Everything Everywhere in 3Dcited by 12
  3. 2025Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Contextcited by 10
  4. 2024MSSPlace: Multi-Sensor Place Recognition with Visual and Text Semanticscited by 7
  5. 2025Talk2Event: Grounded Understanding of Dynamic Scenes from Event Camerascited by 6
  6. 2024LidaRefer: Context-aware Outdoor 3D Visual Grounding for Autonomous Drivingcited by 1
  7. 2025A Coarse-to-Fine Approach to Multi-Modality 3D Occupancy Groundingcited by 1
  8. 2025Visual Grounding from Event Camerascited by 1