# Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds

arXiv: https://arxiv.org/abs/2204.10688 · HTML: https://mlsanity.com/paper/2204.10688/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 42 references that are in the map.
- First posted on arXiv 22 Apr 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Scan2Cap: Context-aware Dense Captioning in RGB-D Scans](https://mlsanity.com/paper/2012.02206/) (2020)
- [Group-Free 3D Object Detection via Transformers](https://mlsanity.com/paper/2104.00678/) (2021)
- [Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions](https://mlsanity.com/paper/2004.03967/) (2020)
- [DenseCap: Fully Convolutional Localization Networks for Dense Captioning](https://mlsanity.com/paper/1511.07571/) (2015)
- [Image Captioning: Transforming Objects into Words](https://mlsanity.com/paper/1906.05963/) (2019)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [Dual-Level Collaborative Transformer for Image Captioning](https://mlsanity.com/paper/2101.06462/) (2021)
- [Exploiting Edge-Oriented Reasoning for 3D Point-based Scene Graph Analysis](https://mlsanity.com/paper/2103.05558/) (2021)
- [Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Network](https://mlsanity.com/paper/2012.07061/) (2020)
- [Dense Captioning with Joint Inference and Visual Context](https://mlsanity.com/paper/1611.06949/) (2016)
- [Dense Relational Captioning: Triple-Stream Networks for Relationship-Based Captioning](https://mlsanity.com/paper/1903.05942/) (2019)
- [Context and Attribute Grounded Dense Captioning](https://mlsanity.com/paper/1904.01410/) (2019)

## Built on by (most cited first, 30 of 40)

- [LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning](https://mlsanity.com/paper/2311.18651/) (2023)
- [Grounded 3D-LLM with Referent Tokens](https://mlsanity.com/paper/2405.10370/) (2024)
- [EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding](https://mlsanity.com/paper/2209.14941/) (2022)
- [End-to-End 3D Dense Captioning with Vote2Cap-DETR](https://mlsanity.com/paper/2301.02508/) (2023)
- [3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer](https://mlsanity.com/paper/2501.01163/) (2025)
- [CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes](https://mlsanity.com/paper/2304.06061/) (2023)
- [ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilities](https://mlsanity.com/paper/2407.01525/) (2024)
- [TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenes](https://mlsanity.com/paper/2403.19589/) (2024)
- [Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioning](https://mlsanity.com/paper/2309.02999/) (2023)
- [ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenes](https://mlsanity.com/paper/2212.06250/) (2022)
- [Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding](https://mlsanity.com/paper/2305.10714/) (2023)
- [Contextual Modeling for 3D Dense Captioning on Point Clouds](https://mlsanity.com/paper/2210.03925/) (2022)
- [CityRefer: Geography-aware 3D Visual Grounding Dataset on City-scale Point Cloud Data](https://mlsanity.com/paper/2310.18773/) (2023)
- [Dense Object Grounding in 3D Scenes](https://mlsanity.com/paper/2309.02224/) (2023)
- ["Where am I?" Scene Retrieval with Language](https://mlsanity.com/paper/2404.14565/) (2024)
- [DC-Scene: Data-Centric Learning for 3D Scene Understanding](https://mlsanity.com/paper/2505.15232/) (2025)
- [3D CoCa: Contrastive Learners are 3D Captioners](https://mlsanity.com/paper/2504.09518/) (2025)
- [Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluation](https://mlsanity.com/paper/2310.15676/) (2023)
- [Cross3DVG: Cross-Dataset 3D Visual Grounding on Different RGB-D Scans](https://mlsanity.com/paper/2305.13876/) (2023)
- [A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes](https://mlsanity.com/paper/2403.07469/) (2024)
- [PerLA: Perceptive 3D Language Assistant](https://mlsanity.com/paper/2411.19774/) (2024)
- [Language-Assisted 3D Feature Learning for Semantic Scene Understanding](https://mlsanity.com/paper/2211.14091/) (2022)
- [MoE3D: Mixture of Experts meets Multi-Modal 3D Understanding](https://mlsanity.com/paper/2511.22103/) (2025)
- [Bi-directional Contextual Attention for 3D Dense Captioning](https://mlsanity.com/paper/2408.06662/) (2024)
- [On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation](https://mlsanity.com/paper/2404.08540/) (2024)
- [See It All: Contextualized Late Aggregation for 3D Dense Captioning](https://mlsanity.com/paper/2408.07648/) (2024)
- [ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detail](https://mlsanity.com/paper/2503.17044/) (2025)
- [OE3DIS: Open-Ended 3D Point Cloud Instance Segmentation](https://mlsanity.com/paper/2408.11747/) (2024)
- [ByDeWay: Boost Your multimodal LLM with DEpth prompting in a Training-Free Way](https://mlsanity.com/paper/2507.08679/) (2025)
- [3D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligence](https://mlsanity.com/paper/2601.06496/) (2026)

## Abstract (arXiv)

Dense captioning in 3D point clouds is an emerging vision-and-language task involving object-level 3D scene understanding. Apart from coarse semantic class prediction and bounding box regression as in traditional 3D object detection, 3D dense captioning aims at producing a further and finer instance-level label of natural language description on visual appearance and spatial relations for each scene object of interest. To detect and describe objects in a scene, following the spirit of neural machine translation, we propose a transformer-based encoder-decoder architecture, namely SpaCap3D, to transform objects into descriptions, where we especially investigate the relative spatiality of objects in 3D scenes and design a spatiality-guided encoder via a token-to-token spatial relation learning objective and an object-centric decoder for precise and spatiality-enhanced object caption generation. Evaluated on two benchmark datasets, ScanRefer and ReferIt3D, our proposed SpaCap3D outperforms the baseline method Scan2Cap by 4.94% and 9.61% in CIDEr@0.5IoU, respectively. Our project page with source code and supplementary files is available at https://SpaCap3D.github.io/ .