# Toward Explainable and Fine-Grained 3D Grounding through Referring Textual Phrases

arXiv: https://arxiv.org/abs/2207.01821 · HTML: https://mlsanity.com/paper/2207.01821/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 55 references that are in the map.
- First posted on arXiv 5 Jul 2022.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2012.15409/) (2020)
- [Scan2Cap: Context-aware Dense Captioning in RGB-D Scans](https://mlsanity.com/paper/2012.02206/) (2020)
- [Group-Free 3D Object Detection via Transformers](https://mlsanity.com/paper/2104.00678/) (2021)
- [Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings](https://mlsanity.com/paper/1803.08495/) (2018)
- [InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring](https://mlsanity.com/paper/2103.01128/) (2021)
- [Multi-View Transformer for 3D Visual Grounding](https://mlsanity.com/paper/2204.02174/) (2022)
- [Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds](https://mlsanity.com/paper/2112.08879/) (2021)
- [SAT: 2D Semantics Assisted Training for 3D Visual Grounding](https://mlsanity.com/paper/2105.11450/) (2021)

## Built on by (most cited first, 16 of 16)

- [Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding](https://mlsanity.com/paper/2311.15383/) (2023)
- [3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination](https://mlsanity.com/paper/2406.05132/) (2024)
- [ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenes](https://mlsanity.com/paper/2212.06250/) (2022)
- [M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts](https://mlsanity.com/paper/2312.10763/) (2023)
- [A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directions](https://mlsanity.com/paper/2406.05785/) (2024)
- [3D Vision and Language Pretraining with Large-Scale Synthetic Data](https://mlsanity.com/paper/2407.06084/) (2024)
- [CityRefer: Geography-aware 3D Visual Grounding Dataset on City-scale Point Cloud Data](https://mlsanity.com/paper/2310.18773/) (2023)
- [Dense Object Grounding in 3D Scenes](https://mlsanity.com/paper/2309.02224/) (2023)
- [CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding](https://mlsanity.com/paper/2310.06214/) (2023)
- [SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes](https://mlsanity.com/paper/2507.07781/) (2025)
- [Cross3DVG: Cross-Dataset 3D Visual Grounding on Different RGB-D Scans](https://mlsanity.com/paper/2305.13876/) (2023)
- [Empowering Large Language Models with 3D Situation Awareness](https://mlsanity.com/paper/2503.23024/) (2025)
- [Data-Efficient 3D Visual Grounding via Order-Aware Referring](https://mlsanity.com/paper/2403.16539/) (2024)
- [R2G: Reasoning to Ground in 3D Scenes](https://mlsanity.com/paper/2408.13499/) (2024)
- [Direction-aware 3D Large Multimodal Models](https://mlsanity.com/paper/2602.19063/) (2026)
- [VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inference](https://mlsanity.com/paper/2608.24063/) (2026)

## Abstract (arXiv)

Recent progress in 3D scene understanding has explored visual grounding (3DVG) to localize a target object through a language description. However, existing methods only consider the dependency between the entire sentence and the target object, ignoring fine-grained relationships between contexts and non-target ones. In this paper, we extend 3DVG to a more fine-grained and interpretable task, called 3D Phrase Aware Grounding (3DPAG). The 3DPAG task aims to localize the target objects in a 3D scene by explicitly identifying all phrase-related objects and then conducting the reasoning according to contextual phrases. To tackle this problem, we manually labeled about 227K phrase-level annotations using a self-developed platform, from 88K sentences of widely used 3DVG datasets, i.e., Nr3D, Sr3D and ScanRefer. By tapping on our datasets, we can extend previous 3DVG methods to the fine-grained phrase-aware scenario. It is achieved through the proposed novel phrase-object alignment optimization and phrase-specific pre-training, boosting conventional 3DVG performance as well. Extensive results confirm significant improvements, i.e., previous state-of-the-art method achieves 3.9%, 3.5% and 4.6% overall accuracy gains on Nr3D, Sr3D and ScanRefer respectively.