mlsanityData to 7 Oct 2026

InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring

arXiv 2103.01128 ↗First posted 1 Mar 2021cs.CVcited by 140

Not on the map. Why?

Abstract (arXiv)

Compared with the visual grounding on 2D images, the natural-language-guided 3D object localization on point clouds is more challenging. In this paper, we propose a new model, named InstanceRefer, to achieve a superior 3D visual grounding through the grounding-by-matching strategy. In practice, our model first predicts the target category from the language descriptions using a simple language classification model. Then, based on the category, our model sifts out a small number of instance candidates (usually less than 20) from the panoptic segmentation of point clouds. Thus, the non-trivial 3D visual grounding task has been effectively re-formulated as a simplified instance-matching problem, considering that instance-level candidates are more rational than the redundant 3D object proposals. Subsequently, for each candidate, we perform the multi-level contextual inference, i.e., referring from instance attribute perception, instance-to-instance relation perception, and instance-to-background global localization perception, respectively. Eventually, the most relevant candidate is selected and localized by ranking confidence scores, which are obtained by the cooperative holistic visual-language feature matching. Experiments confirm that our method outperforms previous state-of-the-arts on ScanRefer online benchmark and Nr3D/Sr3D datasets.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Builds on 40 of its 51 references that are in the map. First posted on arXiv 1 Mar 2021.

Builds on

40 of 51 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  3. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  6. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  8. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  9. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  10. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  11. 2018Panoptic Segmentationcited by 796
  12. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
28 more
  1. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
  2. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  3. 2019RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Cloudscited by 584
  4. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  5. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  6. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 384
  7. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 340
  8. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  9. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  10. 2016ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentationcited by 291
  11. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  12. 2020PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Samplingcited by 238
  13. 2020Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completioncited by 222
  14. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  15. 2021Deep Parametric Continuous Convolutional Neural Networkscited by 161
  16. 2017Pointwise Convolutional Neural Networkscited by 126
  17. 2015Natural Language Object Retrievalcited by 124
  18. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  19. 2018Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networkscited by 102
  20. 2020Graph Structured Network for Image-Text Matchingcited by 93
  21. 2019Improving Referring Expression Grounding with Cross-modal Attention-guided Erasingcited by 88
  22. 2020Graph-Structured Referring Expression Reasoning in The Wildcited by 58
  23. 2019Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methodscited by 55
  24. 2019Learning Cross-modal Context Graph for Visual Groundingcited by 49
  25. 2019Point Cloud Oversegmentation with Graph-Structured Deep Metric Learningcited by 42
  26. 2020Rel3D: A Minimally Contrastive Benchmark for Grounding Spatial Relations in 3Dcited by 30
  27. 2019PyramNet: Point Cloud Pyramid Attention Network and Graph Embedding Module for Classification and Segmentationcited by 6
  28. 2019Dilated Point Convolutions: On the Receptive Field Size of Point Convolutions on 3D Point Cloudscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 18; 2023: 30; 2024: 31; 2025: 29; 2026: 21.

140 papers.

Sort:

By area:

  1. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  2. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  3. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  4. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  5. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  6. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  7. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  8. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  9. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  10. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  11. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  12. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 98
38 more
  1. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88
  2. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  3. 2021Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloudcited by 81
  4. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  5. 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
  6. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  7. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  8. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  9. 2023NS3D: Neuro-Symbolic Grounding of 3D Objects and Relationscited by 57
  10. 2022A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspectivecited by 56
  11. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  12. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  13. 2023Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-trainingcited by 52
  14. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  15. 2024Generating Human Motion in 3D Scenes from Text Descriptionscited by 49
  16. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  17. 20213D Question Answeringcited by 41
  18. 2022Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Groundingcited by 41
  19. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  20. 2022Text2Pos: Text-to-Point-Cloud Cross-Modal Localizationcited by 40
  21. 20233D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentationcited by 36
  22. 2023Text2Loc: 3D Point Cloud Localization from Natural Languagecited by 35
  23. 2022ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenescited by 32
  24. 2023Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Groundingcited by 32
  25. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  26. 2021PartGlot: Learning Shape Part Segmentation from Language Reference Gamescited by 30
  27. 2024MiKASA: Multi-Key-Anchor & Scene-Aware Transformer for 3D Visual Groundingcited by 29
  28. 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Modelcited by 27
  29. 2022Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baselinecited by 25
  30. 2023Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understandingcited by 25
  31. 20233DRP-Net: 3D Relative Position-aware Network for 3D Visual Groundingcited by 25
  32. 2024RefMask3D: Language-Guided Transformer for 3D Referring Segmentationcited by 25
  33. 2022Graph Enhanced Contrastive Learning for Radiology Findings Summarizationcited by 24
  34. 2023Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detectioncited by 24
  35. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 24
  36. 2025Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3Dcited by 23
  37. 2023Mono3DVG: 3D Visual Grounding in Monocular Imagescited by 22
  38. 2023Text to Point Cloud Localization with Relation-Enhanced Transformercited by 20

50 most cited of 140 All 140 (JSON)