mlsanityData to 14 Oct 2026

D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding

arXiv 2112.01551 ↗First posted 2 Dec 2021cs.CVcited by 52

Not on the map. Why?

Abstract (arXiv)

Recent studies on dense captioning and visual grounding in 3D have achieved impressive results. Despite developments in both areas, the limited amount of available 3D vision-language data causes overfitting issues for 3D visual grounding and 3D dense captioning methods. Also, how to discriminatively describe objects in complex 3D environments is not fully studied yet. To address these challenges, we present D3Net, an end-to-end neural speaker-listener architecture that can detect, describe and discriminate. Our D3Net unifies dense captioning and visual grounding in 3D in a self-critical manner. This self-critical property of D3Net also introduces discriminability during object caption generation and enables semi-supervised training on ScanNet data with partially annotated descriptions. Our method outperforms SOTA methods in both tasks on the ScanRefer dataset, surpassing the SOTA 3D dense captioning method by a significant margin.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 53 of its 65 references that are in the map. First posted on arXiv 2 Dec 2021.

Builds on

53 of 65 references.

References · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  12. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
38 more
  1. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  3. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  4. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  5. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  6. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  7. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  8. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  9. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  10. 2016Self-critical Sequence Training for Image Captioningcited by 547
  11. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 484
  12. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  13. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  14. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  15. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  16. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  17. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 321
  18. 2016Segmentation from Natural Language Expressionscited by 257
  19. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 232
  20. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  21. 20183D-SIS: 3D Semantic Instance Segmentation of RGB-D Scanscited by 217
  22. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  23. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  24. 2016Generating Visual Explanationscited by 158
  25. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 141
  26. 2021Hierarchical Aggregation for 3D Instance Segmentationcited by 127
  27. 2015Natural Language Object Retrievalcited by 124
  28. 2020H3DNet: 3D Object Detection Using Hybrid Geometric Primitivescited by 123
  29. 20183DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentationcited by 118
  30. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 89
  31. 2016A Joint Speaker-Listener-Reinforcer Model for Referring Expressionscited by 74
  32. 2018Discriminability objective for training descriptive captionscited by 67
  33. 2019ShapeGlot: Learning Language for Shape Differentiationcited by 54
  34. 2017Comprehension-guided referring expressionscited by 52
  35. 2018Recurrent Fusion Network for Image Captioningcited by 43
  36. 2021Language Grounding with 3D Objectscited by 43
  37. 2016Dense Captioning with Joint Inference and Visual Contextcited by 40
  38. 2018Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Datacited by 37

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 7; 2024: 13; 2025: 17; 2026: 13.

52 papers.

Sort:

By area:

  1. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Modelscited by 235
  2. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  3. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 120
  4. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 70
  5. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  6. 2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
  7. 20253D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understandingcited by 47
  8. 2023SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priorscited by 43
  9. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  10. 20253D Aware Region Prompted Vision Language Modelcited by 33
  11. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  12. 2022Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baselinecited by 25
38 more
  1. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 25
  2. 2024RefMask3D: Language-Guided Transformer for 3D Referring Segmentationcited by 25
  3. 2023Mono3DVG: 3D Visual Grounding in Monocular Imagescited by 22
  4. 2024Collaborative Novel Object Discovery and Box-Guided Cross-Modal Alignment for Open-Vocabulary 3D Object Detectioncited by 17
  5. 2025DC-Scene: Data-Centric Learning for 3D Scene Understandingcited by 15
  6. 2023Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluationcited by 12
  7. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Groundingcited by 12
  8. 2024Transcrib3D: 3D Referring Expression Resolution through Large Language Modelscited by 11
  9. 2024Task-oriented Sequential Grounding and Navigation in 3D Scenescited by 9
  10. 2024PerLA: Perceptive 3D Language Assistantcited by 9
  11. 2025Ges3ViG: Incorporating Pointing Gestures into Language-Based 3D Visual Grounding for Embodied Reference Understandingcited by 8
  12. 2026Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Mapscited by 8
  13. 2025MoE3D: Mixture of Experts meets Multi-Modal 3D Understandingcited by 6
  14. 2024Bi-directional Contextual Attention for 3D Dense Captioningcited by 5
  15. 2024SeCG: Semantic-Enhanced 3D Visual Grounding via Cross-modal Graph Attentioncited by 4
  16. 2024See It All: Contextualized Late Aggregation for 3D Dense Captioningcited by 4
  17. 2024Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attentioncited by 4
  18. 2025ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detailcited by 4
  19. 2025From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenescited by 4
  20. 2026Let Geometry GUIDE: Layer-wise Unrolling of Geometric Priors in Multimodal LLMscited by 4
  21. 2023Weakly-Supervised 3D Visual Grounding based on Visual Language Alignmentcited by 3
  22. 2026Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLMcited by 2
  23. 2025A Coarse-to-Fine Approach to Multi-Modality 3D Occupancy Groundingcited by 1
  24. 2025Text-Scene: A Scene-to-Language Parsing Framework for 3D Scene Understandingcited by 1
  25. 2026Direction-aware 3D Large Multimodal Modelscited by 1
  26. 2026CustomTex: High-fidelity Indoor Scene Texturing via Multi-Reference Customizationcited by 1
  27. 2026Scalable Object Relation Encoding for Better 3D Spatial Reasoning in Large Language Modelscited by 1
  28. 2026Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Groundingcited by 1
  29. 20263D-IDE: 3D Implicit Depth Emergentcited by 1
  30. 2024PD-APE: A Parallel Decoding Framework with Adaptive Position Encoding for 3D Visual Groundingcited by 0
  31. 2025Multi-Object Grounding via Hierarchical Contrastive Siamese Transformerscited by 0
  32. 2025LSVG: Language-Guided Scene Graphs with 2D-Assisted Multi-Modal Encoding for 3D Visual Groundingcited by 0
  33. 2025A Study of the Framework and Real-World Applications of Language Embedding for 3D Scene Understandingcited by 0
  34. 2025B2N3D: Progressive Learning from Binary to N-ary Relationships for 3D Object Groundingcited by 0
  35. 2025Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomycited by 0
  36. 2026Curvature-Aware Captioning:Leveraging Geodesic Attention for 3D Scene Understandingcited by 0
  37. 2026Holo-Captioning: Toward the Text Equivalent of 3D Scenescited by 0
  38. 2026PVCap: Towards Accurate 3D Dense Captioning via PseudoCap and VoxelCapNetcited by 0

50 most cited of 52 All 52 (JSON)