mlsanityData to 7 Oct 2026

ScanERU: Interactive 3D Visual Grounding based on Embodied Reference Understanding

arXiv 2303.13186 ↗First posted 23 Mar 2023cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Aiming to link natural language descriptions to specific regions in a 3D scene represented as 3D point clouds, 3D visual grounding is a very fundamental task for human-robot interaction. The recognition errors can significantly impact the overall accuracy and then degrade the operation of AI systems. Despite their effectiveness, existing methods suffer from the difficulty of low recognition accuracy in cases of multiple adjacent objects with similar appearances.To address this issue, this work intuitively introduces the human-robot interaction as a cue to facilitate the development of 3D visual grounding. Specifically, a new task termed Embodied Reference Understanding (ERU) is first designed for this concern. Then a new dataset called ScanERU is constructed to evaluate the effectiveness of this idea. Different from existing datasets, our ScanERU is the first to cover semi-synthetic scene integration with textual, real-world visual, and synthetic gestural information. Additionally, this paper formulates a heuristic framework based on attention mechanisms and human body movements to enlighten the research of ERU. Experimental results demonstrate the superiority of the proposed method, especially in the recognition of multiple identical objects. Our codes and dataset are ready to be available publicly.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 32 of its 53 references that are in the map. First posted on arXiv 23 Mar 2023.

Builds on

32 of 53 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  3. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  4. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2016Modeling Context in Referring Expressionscited by 1,054
  7. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  8. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  9. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  10. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  11. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  12. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
20 more
  1. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  2. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  3. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 139
  4. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  5. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 137
  6. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  7. 2016GuessWhat?! Visual object discovery through multi-modal dialoguecited by 112
  8. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  9. 2018Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networkscited by 102
  10. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88
  11. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  12. 2021Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloudcited by 81
  13. 2020Referring Expression Comprehension: A Survey of Methods and Datasetscited by 78
  14. 2019CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressionscited by 73
  15. 2022Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoningcited by 69
  16. 2022Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Groundingcited by 41
  17. 2020Point2Skeleton: Learning Skeletal Representations from Point Cloudscited by 37
  18. 2021Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Imagescited by 33
  19. 2021YouRefIt: Embodied Reference Understanding with Language and Gesturecited by 32
  20. 2022Understanding Embodied Reference with Touch-Line Transformercited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 2.

12 papers.

  1. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  2. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 24
  3. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Groundingcited by 12
  4. 2024RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentationcited by 11
  5. 2024Task-oriented Sequential Grounding and Navigation in 3D Scenescited by 9
  6. 2025Ges3ViG: Incorporating Pointing Gestures into Language-Based 3D Visual Grounding for Embodied Reference Understandingcited by 7
  7. 2024Talk to Parallel LiDARs: A Human-LiDAR Interaction Method Based on 3D Visual Groundingcited by 5
  8. 2025From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenescited by 4
  9. 2026MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialoguecited by 1
  10. 2025CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understandingcited by 0
  11. 2025A Multimodal Depth-Aware Method For Embodied Reference Understandingcited by 0
  12. 2026Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guaranteescited by 0