mlsanityData to 7 Oct 2026

End-to-End 3D Dense Captioning with Vote2Cap-DETR

arXiv 2301.02508 ↗First posted 6 Jan 2023cs.CVcited by 77

Not on the map. Why?

Cited by 28 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

3D dense captioning aims to generate multiple captions localized with their associated object regions. Existing methods follow a sophisticated ``detect-then-describe'' pipeline equipped with numerous hand-crafted components. However, these hand-crafted components would yield suboptimal performance given cluttered object spatial and class distributions among different scenes. In this paper, we propose a simple-yet-effective transformer framework Vote2Cap-DETR based on recent popular DEtection TRansformer (DETR). Compared with prior arts, our framework has several appealing advantages: 1) Without resorting to numerous hand-crafted components, our method is based on a full transformer encoder-decoder architecture with a learnable vote query driven object decoder, and a caption decoder that produces the dense captions in a set-prediction manner. 2) In contrast to the two-stage scheme, our method can perform detection and captioning in one-stage. 3) Without bells and whistles, extensive experiments on two commonly used datasets, ScanRefer and Nr3D, demonstrate that our Vote2Cap-DETR surpasses current state-of-the-arts by 11.13% and 7.11% in [email protected], respectively. Codes will be released soon.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 37 of its 56 references that are in the map. First posted on arXiv 6 Jan 2023.

Builds on

37 of 56 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  6. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  10. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  11. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  12. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
25 more
  1. 2016Self-critical Sequence Training for Image Captioningcited by 544
  2. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  3. 2021Conditional DETR for Fast Training Convergencecited by 422
  4. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  5. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  6. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  7. 2019Attention on Attention for Image Captioningcited by 289
  8. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  9. 2021Group-Free 3D Object Detection via Transformerscited by 215
  10. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  11. 2020X-Linear Attention Networks for Image Captioningcited by 198
  12. 2021Fast Convergence of DETR with Spatially Modulated Co-Attentioncited by 176
  13. 2022DETRs with Hybrid Matchingcited by 149
  14. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  15. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  16. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  17. 2021CPTR: Full Transformer Network for Image Captioningcited by 55
  18. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  19. 2021SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videoscited by 40
  20. 2022Spatiality-guided Transformer for 3D Dense Captioning on Point Cloudscited by 40
  21. 2019TightCap: 3D Human Shape Capture with Clothing Tightness Fieldcited by 25
  22. 2022Conditional DETR V2: Efficient Detection Transformer with Box Queriescited by 21
  23. 2022Contextual Modeling for 3D Dense Captioning on Point Cloudscited by 18
  24. 2021Point Cloud Instance Segmentation with Semi-supervised Bounding-Box Miningcited by 15
  25. 2022DETR++: Taming Your Multi-Scale Detection Transformercited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 22; 2025: 25; 2026: 17.

77 papers.

Sort:

By area:

  1. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  2. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  3. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  4. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  5. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  6. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  7. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  8. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  9. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  10. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  11. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  12. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
38 more
  1. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  2. 2024Multi-modal Situated Reasoning in 3D Scenescited by 50
  3. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  4. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  5. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  6. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  7. 2023Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioningcited by 38
  8. 2024LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferencescited by 36
  9. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  10. 2023Dense Object Grounding in 3D Scenescited by 15
  11. 2025DC-Scene: Data-Centric Learning for 3D Scene Understandingcited by 15
  12. 2023T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistencycited by 14
  13. 20253D CoCa: Contrastive Learners are 3D Captionerscited by 13
  14. 2023Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluationcited by 12
  15. 2025DenseGrounding: Improving Dense Language-Vision Semantics for Ego-Centric 3D Visual Groundingcited by 12
  16. 2025GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyondcited by 12
  17. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answeringcited by 11
  18. 2023Cross3DVG: Cross-Dataset 3D Visual Grounding on Different RGB-D Scanscited by 10
  19. 2023Leveraging VLM-Based Pipelines to Annotate 3D Objectscited by 10
  20. 2024A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenescited by 10
  21. 2024Multimodal 3D Reasoning Segmentation with Complex Scenescited by 8
  22. 2024PerLA: Perceptive 3D Language Assistantcited by 8
  23. 2025Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMscited by 8
  24. 2025HIS-GPT: Towards 3D Human-In-Scene Multimodal Understandingcited by 6
  25. 2025MoE3D: Mixture of Experts meets Multi-Modal 3D Understandingcited by 6
  26. 2024Bi-directional Contextual Attention for 3D Dense Captioningcited by 5
  27. 20243D Spatial Understanding in MLLMs: Disambiguation and Evaluationcited by 5
  28. 2025Empowering Large Language Models with 3D Situation Awarenesscited by 5
  29. 2025HouseTour: A Virtual Real Estate A(I)gentcited by 5
  30. 2025CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoningcited by 5
  31. 2025ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detailcited by 4
  32. 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Expertscited by 4
  33. 2024OE3DIS: Open-Ended 3D Point Cloud Instance Segmentationcited by 3
  34. 2024MTA: Multimodal Task Alignment for BEV Perception and Captioningcited by 3
  35. 20263D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligencecited by 3
  36. 2025Exploring 3D Reasoning-Driven Planning: From Implicit Human Intentions to Route-Aware Activity Planningcited by 2
  37. 2026Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLMcited by 2
  38. 2026Let Geometry GUIDE: Layer-wise Unrolling of Geometric Priors in Multimodal LLMscited by 2

50 most cited of 77 All 77 (JSON)