mlsanityData to 7 Oct 2026

UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding

arXiv 2212.00836 ↗First posted 1 Dec 2022cs.CVcited by 69

Not on the map. Why?

Abstract (arXiv)

Performing 3D dense captioning and visual grounding requires a common and shared understanding of the underlying multimodal relationships. However, despite some previous attempts on connecting these two related tasks with highly task-specific neural modules, it remains understudied how to explicitly depict their shared nature to learn them simultaneously. In this work, we propose UniT3D, a simple yet effective fully unified transformer-based architecture for jointly solving 3D visual grounding and dense captioning. UniT3D enables learning a strong multimodal representation across the two tasks through a supervised joint pre-training scheme with bidirectional and seq-to-seq objectives. With a generic architecture design, UniT3D allows expanding the pre-training scope to more various training sources such as the synthesized data from 2D prior knowledge to benefit 3D vision-language tasks. Extensive experiments and analysis demonstrate that UniT3D obtains significant gains for 3D dense captioning and visual grounding.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 43 of its 52 references that are in the map. First posted on arXiv 1 Dec 2022.

Builds on

43 of 52 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
31 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  2. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  3. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  4. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  7. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  8. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  9. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  10. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  11. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  12. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  13. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  14. 20183D-SIS: 3D Semantic Instance Segmentation of RGB-D Scanscited by 216
  15. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  16. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  17. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  18. 20203D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentationcited by 140
  19. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  20. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  21. 2020ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votescited by 119
  22. 20183DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentationcited by 118
  23. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  24. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  25. 20193D Instance Segmentation via Multi-Task Metric Learningcited by 79
  26. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  27. 2021Back-tracing Representative Points for Voting-based 3D Object Detection in Point Cloudscited by 67
  28. 20213D-to-2D Distillation for Indoor Scene Parsingcited by 47
  29. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  30. 2019RevealNet: Seeing Behind Objects in RGB-D Scanscited by 30
  31. 2021Voint Cloud: Multi-View Point Cloud Representation for 3D Understandingcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 15; 2024: 24; 2025: 19; 2026: 10.

69 papers.

Sort:

By area:

  1. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Modelscited by 234
  2. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  3. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  4. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  5. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  6. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  7. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  8. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  9. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  10. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  11. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  12. 2024Multi-modal Situated Reasoning in 3D Scenescited by 50
38 more
  1. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  2. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  3. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  4. 2023SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priorscited by 43
  5. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  6. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  7. 2023Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioningcited by 38
  8. 2022ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenescited by 32
  9. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  10. 2023Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understandingcited by 25
  11. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 24
  12. 2024Collaborative Novel Object Discovery and Box-Guided Cross-Modal Alignment for Open-Vocabulary 3D Object Detectioncited by 17
  13. 2023Dense Object Grounding in 3D Scenescited by 15
  14. 20253D CoCa: Contrastive Learners are 3D Captionerscited by 13
  15. 2024Grounding 3D Scene Affordance From Egocentric Interactionscited by 11
  16. 2023Cross3DVG: Cross-Dataset 3D Visual Grounding on Different RGB-D Scanscited by 10
  17. 2024CT-AGRG: Automated Abnormality-Guided Report Generation from 3D Chest CT Volumescited by 10
  18. 2025Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understandingcited by 10
  19. 2023Uni3DL: Unified Model for 3D and Language Understandingcited by 9
  20. 2024Task-oriented Sequential Grounding and Navigation in 3D Scenescited by 9
  21. 2023WildRefer: 3D Object Localization in Large-scale Dynamic Scenes with Multi-modal Visual Data and Natural Languagecited by 8
  22. 2024Multimodal 3D Reasoning Segmentation with Complex Scenescited by 8
  23. 2024PerLA: Perceptive 3D Language Assistantcited by 8
  24. 20253EED: Ground Everything Everywhere in 3Dcited by 8
  25. 2026Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Mapscited by 8
  26. 2025ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Groundingcited by 7
  27. 2025Ges3ViG: Incorporating Pointing Gestures into Language-Based 3D Visual Grounding for Embodied Reference Understandingcited by 7
  28. 2025SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenescited by 7
  29. 2025UniPre3D: Unified Pre-training of 3D Point Cloud Models with Cross-Modal Gaussian Splattingcited by 6
  30. 2023Generating Context-Aware Natural Answers for Questions in 3D Scenescited by 5
  31. 2024Unified Scene Representation and Reconstruction for 3D Large Language Modelscited by 5
  32. 20243D-AVS: LiDAR-based 3D Auto-Vocabulary Segmentationcited by 5
  33. 2024Bi-directional Contextual Attention for 3D Dense Captioningcited by 5
  34. 2024See It All: Contextualized Late Aggregation for 3D Dense Captioningcited by 4
  35. 2024Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attentioncited by 4
  36. 2025ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detailcited by 4
  37. 2025From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenescited by 4
  38. 2025GroundFlow: A Plug-in Module for Temporal Reasoning on 3D Point Cloud Sequential Groundingcited by 4

50 most cited of 69 All 69 (JSON)