# Bi-directional Contextual Attention for 3D Dense Captioning

arXiv: https://arxiv.org/abs/2408.06662 · HTML: https://mlsanity.com/paper/2408.06662/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 13 Aug 2024.

## Builds on (most cited first, 25 of 25)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [An End-to-End Transformer Model for 3D Object Detection](https://mlsanity.com/paper/2109.08141/) (2021)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [Scan2Cap: Context-aware Dense Captioning in RGB-D Scans](https://mlsanity.com/paper/2012.02206/) (2020)
- [A Comprehensive Survey of Deep Learning for Image Captioning](https://mlsanity.com/paper/1810.04020/) (2018)
- [End-to-End 3D Dense Captioning with Vote2Cap-DETR](https://mlsanity.com/paper/2301.02508/) (2023)
- [X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning](https://mlsanity.com/paper/2203.00843/) (2022)
- [UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding](https://mlsanity.com/paper/2212.00836/) (2022)
- [D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding](https://mlsanity.com/paper/2112.01551/) (2021)
- [MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenes](https://mlsanity.com/paper/2203.05203/) (2022)
- [Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds](https://mlsanity.com/paper/2204.10688/) (2022)
- [Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioning](https://mlsanity.com/paper/2309.02999/) (2023)
- [Contextual Modeling for 3D Dense Captioning on Point Clouds](https://mlsanity.com/paper/2210.03925/) (2022)

## Built on by (most cited first, 5 of 5)

- [3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer](https://mlsanity.com/paper/2501.01163/) (2025)
- [3D CoCa: Contrastive Learners are 3D Captioners](https://mlsanity.com/paper/2504.09518/) (2025)
- [3D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligence](https://mlsanity.com/paper/2601.06496/) (2026)
- [Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understanding](https://mlsanity.com/paper/2511.21191/) (2025)
- [Curvature-Aware Captioning:Leveraging Geodesic Attention for 3D Scene Understanding](https://mlsanity.com/paper/2605.08808/) (2026)

## Abstract (arXiv)

3D dense captioning is a task involving the localization of objects and the generation of descriptions for each object in a 3D scene. Recent approaches have attempted to incorporate contextual information by modeling relationships with object pairs or aggregating the nearest neighbor features of an object. However, the contextual information constructed in these scenarios is limited in two aspects: first, objects have multiple positional relationships that exist across the entire global scene, not only near the object itself. Second, it faces with contradicting objectives--where localization and attribute descriptions are generated better with tight localization, while descriptions involving global positional relations are generated better with contextualized features of the global scene. To overcome this challenge, we introduce BiCA, a transformer encoder-decoder pipeline that engages in 3D dense captioning for each object with Bi-directional Contextual Attention. Leveraging parallelly decoded instance queries for objects and context queries for non-object contexts, BiCA generates object-aware contexts, where the contexts relevant to each object is summarized, and context-aware objects, where the objects relevant to the summarized object-aware contexts are aggregated. This extension relieves previous methods from the contradicting objectives, enhancing both localization performance and enabling the aggregation of contextual features throughout the global scene; thus improving caption generation performance simultaneously. Extensive experiments on two of the most widely-used 3D dense captioning datasets demonstrate that our proposed method achieves a significant improvement over prior methods.