mlsanityData to 13 Oct 2026

Language Conditioned Spatial Relation Reasoning for 3D Object Grounding

arXiv 2211.09646 ↗First posted 17 Nov 2022cs.CVcited by 121

Not on the map. Why?

Cited by 26 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Localizing objects in 3D scenes based on natural language requires understanding and reasoning about spatial relations. In particular, it is often crucial to distinguish similar objects referred by the text, such as "the left most chair" and "a chair next to the window". In this work we propose a language-conditioned transformer model for grounding 3D objects and their spatial relations. To this end, we design a spatial self-attention layer that accounts for relative distances and orientations between objects in input 3D point clouds. Training such a layer with visual and language inputs enables to disambiguate spatial relations and to localize objects referred by the text. To facilitate the cross-modal learning of relations, we further propose a teacher-student approach where the teacher model is first trained using ground-truth object labels, and then helps to train a student model using point cloud inputs. We perform ablation studies showing advantages of our approach. We also demonstrate our model to significantly outperform the state of the art on the challenging Nr3D, Sr3D and ScanRefer 3D object grounding datasets.

Key facts. Cited by 121 papers in the map (data through 13 Oct 2026). Builds on 39 of its 56 references that are in the map. First posted on arXiv 17 Nov 2022.

Builds on

39 of 56 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  5. 2017Graph Attention Networkscited by 8,888
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,389
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,096
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,649
  9. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,222
  10. 2017Decoupled Weight Decay Regularizationcited by 1,788
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,157
  12. 2016Modeling Context in Referring Expressionscited by 1,057
27 more
  1. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 898
  2. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 815
  3. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  4. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  5. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  6. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  7. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 430
  8. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  9. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  10. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 319
  11. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  12. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  13. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  14. 2021Group-Free 3D Object Detection via Transformerscited by 215
  15. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 207
  16. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 188
  17. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  18. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  19. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  20. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 120
  21. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  22. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 98
  23. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88
  24. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  25. 2021Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloudcited by 81
  26. 2019CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressionscited by 73
  27. 2020Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehensioncited by 50

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 11; 2024: 32; 2025: 52; 2026: 26.

121 papers.

Sort:

By area:

  1. 2023An Embodied Generalist Agent in 3D Worldcited by 357
  2. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 202
  3. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 179
  4. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 171
  5. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  6. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 132
  7. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  8. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 129
  9. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 113
  10. 2024Grounded 3D-LLM with Referent Tokenscited by 112
  11. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  12. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
38 more
  1. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 68
  2. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  3. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 66
  4. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  5. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  6. 2023PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulationcited by 56
  7. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  8. 2024Multi-modal Situated Reasoning in 3D Scenescited by 50
  9. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  10. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  11. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  12. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  13. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  14. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  15. 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Modelcited by 27
  16. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 26
  17. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 26
  18. 20233DRP-Net: 3D Relative Position-aware Network for 3D Visual Groundingcited by 25
  19. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 24
  20. 2025Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3Dcited by 23
  21. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 22
  22. 20243D Vision and Language Pretraining with Large-Scale Synthetic Datacited by 21
  23. 2024Multi-branch Collaborative Learning Network for 3D Visual Groundingcited by 20
  24. 2024Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehensioncited by 18
  25. 20243D-GRES: Generalized 3D Referring Expression Segmentationcited by 18
  26. 2023Dense Object Grounding in 3D Scenescited by 15
  27. 2025Unifying 2D and 3D Vision-Language Understandingcited by 15
  28. 2023CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Groundingcited by 13
  29. 2025CrossOver: 3D Scene Cross-Modal Alignmentcited by 12
  30. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Groundingcited by 12
  31. 2024RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentationcited by 11
  32. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answeringcited by 11
  33. 2023Cross3DVG: Cross-Dataset 3D Visual Grounding on Different RGB-D Scanscited by 10
  34. 2024Transcrib3D: 3D Referring Expression Resolution through Large Language Modelscited by 10
  35. 2025AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referringcited by 10
  36. 2025SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Modelscited by 10
  37. 2024Task-oriented Sequential Grounding and Navigation in 3D Scenescited by 9
  38. 2025DSPNet: Dual-vision Scene Perception for Robust 3D Question Answeringcited by 9

50 most cited of 121 All 121 (JSON)