mlsanityData to 14 Oct 2026

Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds

arXiv 2204.10688 ↗First posted 22 Apr 2022cs.CVcited by 40

Not on the map. Why?

Abstract (arXiv)

Dense captioning in 3D point clouds is an emerging vision-and-language task involving object-level 3D scene understanding. Apart from coarse semantic class prediction and bounding box regression as in traditional 3D object detection, 3D dense captioning aims at producing a further and finer instance-level label of natural language description on visual appearance and spatial relations for each scene object of interest. To detect and describe objects in a scene, following the spirit of neural machine translation, we propose a transformer-based encoder-decoder architecture, namely SpaCap3D, to transform objects into descriptions, where we especially investigate the relative spatiality of objects in 3D scenes and design a spatiality-guided encoder via a token-to-token spatial relation learning objective and an object-centric decoder for precise and spatiality-enhanced object caption generation. Evaluated on two benchmark datasets, ScanRefer and ReferIt3D, our proposed SpaCap3D outperforms the baseline method Scan2Cap by 4.94% and 9.61% in [email protected], respectively. Our project page with source code and supplementary files is available at https://SpaCap3D.github.io/ .

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 24 of its 42 references that are in the map. First posted on arXiv 22 Apr 2022.

Builds on

24 of 42 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  9. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  10. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  11. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  12. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
12 more
  1. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  2. 2021Group-Free 3D Object Detection via Transformerscited by 215
  3. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 208
  4. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  5. 2019Image Captioning: Transforming Objects into Wordscited by 171
  6. 2018Scan2CAD: Learning CAD Model Alignment in RGB-D Scanscited by 142
  7. 2021Dual-Level Collaborative Transformer for Image Captioningcited by 71
  8. 2021Exploiting Edge-Oriented Reasoning for 3D Point-based Scene Graph Analysiscited by 49
  9. 2020Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Networkcited by 48
  10. 2016Dense Captioning with Joint Inference and Visual Contextcited by 40
  11. 2019Dense Relational Captioning: Triple-Stream Networks for Relationship-Based Captioningcited by 35
  12. 2019Context and Attribute Grounded Dense Captioningcited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 9; 2024: 12; 2025: 7; 2026: 7.

40 papers.

Sort:

By area:

  1. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  2. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  3. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 99
  4. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 78
  5. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 68
  6. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  7. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  8. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  9. 2023Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioningcited by 39
  10. 2022ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenescited by 32
  11. 2023Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understandingcited by 25
  12. 2022Contextual Modeling for 3D Dense Captioning on Point Cloudscited by 18
28 more
  1. 2023CityRefer: Geography-aware 3D Visual Grounding Dataset on City-scale Point Cloud Datacited by 18
  2. 2023Dense Object Grounding in 3D Scenescited by 15
  3. 2024"Where am I?" Scene Retrieval with Languagecited by 15
  4. 2025DC-Scene: Data-Centric Learning for 3D Scene Understandingcited by 15
  5. 20253D CoCa: Contrastive Learners are 3D Captionerscited by 13
  6. 2023Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluationcited by 12
  7. 2023Cross3DVG: Cross-Dataset 3D Visual Grounding on Different RGB-D Scanscited by 10
  8. 2024A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenescited by 10
  9. 2024PerLA: Perceptive 3D Language Assistantcited by 9
  10. 2022Language-Assisted 3D Feature Learning for Semantic Scene Understandingcited by 6
  11. 2025MoE3D: Mixture of Experts meets Multi-Modal 3D Understandingcited by 6
  12. 2024Bi-directional Contextual Attention for 3D Dense Captioningcited by 5
  13. 2024On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimationcited by 4
  14. 2024See It All: Contextualized Late Aggregation for 3D Dense Captioningcited by 4
  15. 2025ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detailcited by 4
  16. 2024OE3DIS: Open-Ended 3D Point Cloud Instance Segmentationcited by 3
  17. 2025ByDeWay: Boost Your multimodal LLM with DEpth prompting in a Training-Free Waycited by 3
  18. 20263D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligencecited by 3
  19. 2026Point Cloud as a Foreign Language for Multi-modal Large Language Modelcited by 2
  20. 2022Novel 3D Scene Understanding Applications From Recurrence in a Single Imagecited by 1
  21. 2026Direction-aware 3D Large Multimodal Modelscited by 1
  22. 2024Rethinking 3D Dense Caption and Visual Grounding in A Unified Framework through Prompt-based Localizationcited by 0
  23. 20243D Scene Graph Guided Vision-Language Pre-trainingcited by 0
  24. 2025Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understandingcited by 0
  25. 2026Curvature-Aware Captioning:Leveraging Geodesic Attention for 3D Scene Understandingcited by 0
  26. 2026Semantic-Aware Guided Drone Exploration for Language-Conditioned 3D Indoor Mappingcited by 0
  27. 2026PVCap: Towards Accurate 3D Dense Captioning via PseudoCap and VoxelCapNetcited by 0
  28. 2026Explainable and Resource-Efficient Spatial Reasoning in Multimodal LLMs for Decision-Critical Applicationscited by 0