mlsanityData to 7 Oct 2026

LanguageRefer: Spatial-Language Model for 3D Visual Grounding

arXiv 2107.03438 ↗First posted 7 Jul 2021cs.RO cs.CL cs.CVcited by 88

Not on the map. Why?

Abstract (arXiv)

For robots to understand human instructions and perform meaningful tasks in the near future, it is important to develop learned models that comprehend referential language to identify common objects in real-world 3D scenes. In this paper, we introduce a spatial-language model for a 3D visual grounding problem. Specifically, given a reconstructed 3D scene in the form of point clouds with 3D bounding boxes of potential object candidates, and a language utterance referring to a target object in the scene, our model successfully identifies the target object from a set of potential candidates. Specifically, LanguageRefer uses a transformer-based architecture that combines spatial embedding from bounding boxes with fine-tuned language embeddings from DistilBert to predict the target object. We show that it performs competitively on visio-linguistic datasets proposed by ReferIt3D. Further, we analyze its spatial reasoning task performance decoupled from perception noise, the accuracy of view-dependent utterances, and viewpoint annotations for potential robotics applications.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Builds on 40 of its 52 references that are in the map. First posted on arXiv 7 Jul 2021.

Builds on

40 of 52 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  7. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  10. 2016Modeling Context in Referring Expressionscited by 1,054
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
28 more
  1. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  2. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  3. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  4. 2018On Evaluation of Embodied Navigation Agentscited by 526
  5. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  6. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  7. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  8. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  9. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  10. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  11. 2017Embodied Question Answeringcited by 335
  12. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  13. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  14. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  15. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  16. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  17. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  18. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  19. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 115
  20. 2019Dynamic Graph Attention for Referring Expression Comprehensioncited by 111
  21. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  22. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  23. 2021Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloudcited by 81
  24. 2019Learning Cross-modal Context Graph for Visual Groundingcited by 49
  25. 2021Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigationcited by 47
  26. 2020Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Followingcited by 18
  27. 2019Conditional Driving from Natural Language Instructionscited by 17
  28. 2021VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigatorcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 14; 2023: 23; 2024: 22; 2025: 15; 2026: 9.

88 papers.

Sort:

By area:

  1. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  2. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  3. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  4. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 156
  5. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  6. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  7. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  8. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  9. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  10. 2021A Survey of Visual Transformerscited by 117
  11. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  12. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 98
38 more
  1. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  2. 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
  3. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  4. 2023NS3D: Neuro-Symbolic Grounding of 3D Objects and Relationscited by 57
  5. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  6. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
  7. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  8. 2024Generating Human Motion in 3D Scenes from Text Descriptionscited by 49
  9. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  10. 2022Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Groundingcited by 41
  11. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  12. 2023Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioningcited by 38
  13. 2022ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenescited by 32
  14. 2023Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Groundingcited by 32
  15. 2021PartGlot: Learning Shape Part Segmentation from Language Reference Gamescited by 30
  16. 2022Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baselinecited by 25
  17. 20233DRP-Net: 3D Relative Position-aware Network for 3D Visual Groundingcited by 25
  18. 2024RefMask3D: Language-Guided Transformer for 3D Referring Segmentationcited by 25
  19. 2023Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detectioncited by 24
  20. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 24
  21. 2025Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3Dcited by 23
  22. 2024Multi-branch Collaborative Learning Network for 3D Visual Groundingcited by 20
  23. 2022TriCoLo: Trimodal Contrastive Loss for Text to Shape Retrievalcited by 18
  24. 2021Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulationcited by 17
  25. 2024Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehensioncited by 17
  26. 2025TSP3D: Text-guided Sparse Voxel Pruning for Efficient 3D Visual Groundingcited by 17
  27. 2022Toward Explainable and Fine-Grained 3D Grounding through Referring Textual Phrasescited by 16
  28. 2023Dense Object Grounding in 3D Scenescited by 15
  29. 2023CityRefer: Geography-aware 3D Visual Grounding Dataset on City-scale Point Cloud Datacited by 15
  30. 2024"Where am I?" Scene Retrieval with Languagecited by 15
  31. 2025Unifying 2D and 3D Vision-Language Understandingcited by 15
  32. 2023CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Groundingcited by 13
  33. 2023ScanERU: Interactive 3D Visual Grounding based on Embodied Reference Understandingcited by 12
  34. 2023Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluationcited by 12
  35. 2024Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Modelscited by 12
  36. 2022Voxel-informed Language Groundingcited by 11
  37. 2024Grounding 3D Scene Affordance From Egocentric Interactionscited by 11
  38. 2023Narrator: Towards Natural Control of Human-Scene Interaction Generation via Relationship Reasoningcited by 10

50 most cited of 88 All 88 (JSON)