# Fine-Grained Spatial and Verbal Losses for 3D Visual Grounding arXiv: https://arxiv.org/abs/2411.03405 ยท HTML: https://mlsanity.com/paper/2411.03405/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Builds on 35 of its 46 references that are in the map. - First posted on arXiv 5 Nov 2024. ## Builds on (most cited first, 30 of 35) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016) - [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017) - [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019) - [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017) - [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021) - [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015) - [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019) - [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020) - [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015) - [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021) - [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019) - [3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment](https://mlsanity.com/paper/2308.04352/) (2023) - [InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring](https://mlsanity.com/paper/2103.01128/) (2021) - [Multi-View Transformer for 3D Visual Grounding](https://mlsanity.com/paper/2204.02174/) (2022) - [Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds](https://mlsanity.com/paper/2112.08879/) (2021) - [Natural Language Object Retrieval](https://mlsanity.com/paper/1511.04164/) (2015) - [SAT: 2D Semantics Assisted Training for 3D Visual Grounding](https://mlsanity.com/paper/2105.11450/) (2021) - [Language Conditioned Spatial Relation Reasoning for 3D Object Grounding](https://mlsanity.com/paper/2211.09646/) (2022) - [3D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selection](https://mlsanity.com/paper/2204.06272/) (2022) - [LanguageRefer: Spatial-Language Model for 3D Visual Grounding](https://mlsanity.com/paper/2107.03438/) (2021) - [TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding](https://mlsanity.com/paper/2108.02388/) (2021) - [Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloud](https://mlsanity.com/paper/2103.16381/) (2021) - [NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations](https://mlsanity.com/paper/2303.13483/) (2023) - [ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolution](https://mlsanity.com/paper/2303.00246/) (2023) - [Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding](https://mlsanity.com/paper/2211.14241/) (2022) - [3D Instances as 1D Kernels](https://mlsanity.com/paper/2207.07372/) (2022) ## Built on by (most cited first, 1 of 1) - [TrackTeller: Temporal Multimodal 3D Grounding for Behavior-Dependent Object References](https://mlsanity.com/paper/2512.21641/) (2025) ## Abstract (arXiv) 3D visual grounding consists of identifying the instance in a 3D scene which is referred by an accompanying language description. While several architectures have been proposed within the commonly employed grounding-by-selection framework, the utilized losses are comparatively under-explored. In particular, most methods rely on a basic supervised cross-entropy loss on the predicted distribution over candidate instances, which fails to model both spatial relations between instances and the internal fine-grained word-level structure of the verbal referral. Sparse attempts to additionally supervise verbal embeddings globally by learning the class of the referred instance from the description or employing verbo-visual contrast to better separate instance embeddings do not fundamentally lift the aforementioned limitations. Responding to these shortcomings, we introduce two novel losses for 3D visual grounding: a visual-level offset loss on regressed vector offsets from each instance to the ground-truth referred instance and a language-related span loss on predictions for the word-level span of the referred instance in the description. In addition, we equip the verbo-visual fusion module of our new 3D visual grounding architecture AsphaltNet with a top-down bidirectional attentive fusion block, which enables the supervisory signals from our two losses to propagate to the respective converse branches of the network and thus aid the latter to learn context-aware instance embeddings and grounding-aware verbal embeddings. AsphaltNet proposes novel auxiliary losses to aid 3D visual grounding with competitive results compared to the state-of-the-art on the ReferIt3D benchmark.