mlsanityData to 7 Oct 2026

Scan2Cap: Context-aware Dense Captioning in RGB-D Scans

arXiv 2012.02206 ↗First posted 3 Dec 2020cs.CV cs.LG eess.IVcited by 215

Not on the map. Why?

Cited by 75 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce the task of dense captioning in 3D scans from commodity RGB-D sensors. As input, we assume a point cloud of a 3D scene; the expected output is the bounding boxes along with the descriptions for the underlying objects. To address the 3D object detection and description problems, we propose Scan2Cap, an end-to-end trained method, to detect objects in the input scene and describe them in natural language. We use an attention mechanism that generates descriptive tokens while referring to the related components in the local context. To reflect object relations (i.e. relative spatial relations) in the generated captions, we use a message passing graph module to facilitate learning object relation features. Our method can effectively localize and describe 3D objects in scenes from the ScanRefer dataset, outperforming 2D baseline methods by a significant margin (27.61% [email protected]).

Key facts. Cited by 215 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 17; 2023: 32; 2024: 41; 2025: 62; 2026: 54.

215 papers.

Sort:

By area:

  1. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  2. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  3. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  4. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Modelscited by 234
  5. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  6. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  7. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  8. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  9. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  10. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  11. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  12. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
38 more
  1. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  2. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  3. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  4. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  5. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  6. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  7. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  8. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  9. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 98
  10. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  11. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  12. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  13. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 77
  14. 20243DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priorscited by 72
  15. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  16. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  17. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  18. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  19. 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63
  20. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  21. 2024CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attentioncited by 62
  22. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  23. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  24. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  25. 2024Multi-modal Situated Reasoning in 3D Scenescited by 50
  26. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  27. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  28. 2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
  29. 2021Exploiting Edge-Oriented Reasoning for 3D Point-based Scene Graph Analysiscited by 47
  30. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  31. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  32. 2023SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priorscited by 43
  33. 20213D Question Answeringcited by 41
  34. 2022Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Groundingcited by 41
  35. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  36. 2022Spatiality-guided Transformer for 3D Dense Captioning on Point Cloudscited by 40
  37. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  38. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40

50 most cited of 215 All 215 (JSON)