mlsanityData to 7 Oct 2026

SAT: 2D Semantics Assisted Training for 3D Visual Grounding

arXiv 2105.11450 ↗First posted 24 May 2021cs.CVcited by 124

Not on the map. Why?

Abstract (arXiv)

3D visual grounding aims at grounding a natural language description about a 3D scene, usually represented in the form of 3D point clouds, to the targeted object region. Point clouds are sparse, noisy, and contain limited semantic information compared with 2D images. These inherent limitations make the 3D visual grounding problem more challenging. In this study, we propose 2D Semantics Assisted Training (SAT) that utilizes 2D image semantics in the training stage to ease point-cloud-language joint representation learning and assist 3D visual grounding. The main idea is to learn auxiliary alignments between rich, clean 2D object representations and the corresponding objects or mentioned entities in 3D scenes. SAT takes 2D object semantics, i.e., object label, image feature, and 2D geometric feature, as the extra input in training but does not require such inputs during inference. By effectively utilizing 2D semantics in training, our approach boosts the accuracy on the Nr3D dataset from 37.7% to 49.2%, which significantly surpasses the non-SAT baseline with the identical network architecture and inference input. Our approach outperforms the state of the art by large margins on multiple 3D visual grounding datasets, i.e., +10.4% absolute accuracy on Nr3D, +9.9% on Sr3D, and +5.6% on ScanRef.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Builds on 47 of its 58 references that are in the map. First posted on arXiv 24 May 2021.

Builds on

47 of 58 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2016Layer Normalizationcited by 5,579
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  10. 2016Modeling Context in Referring Expressionscited by 1,054
  11. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  12. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
35 more
  1. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  2. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  3. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  4. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  5. 2019DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusioncited by 352
  6. 2017Joint 3D Proposal Generation and Object Detection from View Aggregationcited by 325
  7. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  8. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  9. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  10. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  11. 20183D-SIS: 3D Semantic Instance Segmentation of RGB-D Scanscited by 216
  12. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  13. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 205
  14. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  15. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  16. 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
  17. 2017PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimationcited by 170
  18. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  19. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  20. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 137
  21. 2015Natural Language Object Retrievalcited by 124
  22. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  23. 2020ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votescited by 119
  24. 2020Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentencescited by 116
  25. 2015Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Imagescited by 111
  26. 2019Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQAcited by 107
  27. 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95
  28. 2021Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloudcited by 81
  29. 2019Zero-Shot Grounding of Objects from Natural Language Queriescited by 79
  30. 2017Comprehension-guided referring expressionscited by 52
  31. 2019Grounding-Tracking-Integrationcited by 32
  32. 2021CityFlow-NL: Tracking and Retrieval of Vehicles at City Scale by Natural Language Descriptionscited by 26
  33. 2020Video Object Grounding using Semantic Roles in Language Descriptioncited by 25
  34. 2016Faster R-CNN Features for Instance Searchcited by 14
  35. 2020AttnGrounder: Talking to Cars with Attentioncited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 14; 2023: 31; 2024: 34; 2025: 32; 2026: 9.

124 papers.

Sort:

By area:

  1. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  2. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  3. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  4. 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 157
  5. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  6. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  7. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  8. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  9. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  10. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  11. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  12. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
38 more
  1. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 98
  2. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  3. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88
  4. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  5. 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
  6. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  7. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  8. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  9. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  10. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  11. 2023NS3D: Neuro-Symbolic Grounding of 3D Objects and Relationscited by 57
  12. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  13. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
  14. 2024Generating Human Motion in 3D Scenes from Text Descriptionscited by 49
  15. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  16. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  17. 2023VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloudcited by 42
  18. 20213D Question Answeringcited by 41
  19. 2022Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Groundingcited by 41
  20. 20233D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentationcited by 36
  21. 2023What's Left? Concept Grounding with Logic-Enhanced Foundation Modelscited by 33
  22. 2022ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenescited by 32
  23. 2023Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Groundingcited by 32
  24. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  25. 2024MiKASA: Multi-Key-Anchor & Scene-Aware Transformer for 3D Visual Groundingcited by 29
  26. 2022Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baselinecited by 25
  27. 2023Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understandingcited by 25
  28. 20233DRP-Net: 3D Relative Position-aware Network for 3D Visual Groundingcited by 25
  29. 2024RefMask3D: Language-Guided Transformer for 3D Referring Segmentationcited by 25
  30. 2022Let Images Give You More:Point Cloud Cross-Modal Training for Shape Analysiscited by 24
  31. 2023Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detectioncited by 24
  32. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 24
  33. 2025Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3Dcited by 23
  34. 2023Mono3DVG: 3D Visual Grounding in Monocular Imagescited by 22
  35. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 22
  36. 20243D Vision and Language Pretraining with Large-Scale Synthetic Datacited by 21
  37. 2024Multi-branch Collaborative Learning Network for 3D Visual Groundingcited by 20
  38. 2023Language-Guided 3D Object Detection in Point Cloud for Autonomous Drivingcited by 19

50 most cited of 124 All 124 (JSON)