mlsanityData to 14 Oct 2026

3D CoCa: Contrastive Learners are 3D Captioners

arXiv 2504.09518 ↗First posted 13 Apr 2025cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

3D captioning, which aims to describe the content of 3D scenes in natural language, remains highly challenging due to the inherent sparsity of point clouds and weak cross-modal alignment in existing methods. To address these challenges, we propose 3D CoCa, a novel unified framework that seamlessly combines contrastive vision-language learning with 3D caption generation in a single architecture. Our approach leverages a frozen CLIP vision-language backbone to provide rich semantic priors, a spatially-aware 3D scene encoder to capture geometric context, and a multi-modal decoder to generate descriptive captions. Unlike prior two-stage methods that rely on explicit object proposals, 3D CoCa jointly optimizes contrastive and captioning objectives in a shared feature space, eliminating the need for external detectors or handcrafted proposals. This joint training paradigm yields stronger spatial reasoning and richer semantic grounding by aligning 3D and textual representations. Extensive experiments on the ScanRefer and Nr3D benchmarks demonstrate that 3D CoCa significantly outperforms current state-of-the-arts by 10.2% and 5.76% in CIDEr at 0.5IoU, respectively. Code will be available at https://github.com/AIGeeksGroup/3DCoCa.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 13 Apr 2025.

Builds on

32 of 49 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  3. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  6. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  7. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 504
  8. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  9. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  10. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  11. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  12. 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
20 more
  1. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  2. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 78
  3. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  4. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  5. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  6. 2021SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videoscited by 40
  7. 2022Spatiality-guided Transformer for 3D Dense Captioning on Point Cloudscited by 40
  8. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  9. 2023Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioningcited by 39
  10. 2019TightCap: 3D Human Shape Capture with Clothing Tightness Fieldcited by 25
  11. 2023Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understandingcited by 25
  12. 2023Take-A-Photo: 3D-to-2D Generative Pre-training of Point Cloud Modelscited by 25
  13. 2022EPCL: Frozen CLIP Transformer is An Efficient Point Cloud Encodercited by 18
  14. 2021Point Cloud Instance Segmentation with Semi-supervised Bounding-Box Miningcited by 15
  15. 2024MSDet: Receptive Field Enhanced Multiscale Detection for Tiny Pulmonary Nodulecited by 14
  16. 2024MedDet: Generative Adversarial Distillation for Efficient Cervical Disc Herniation Detectioncited by 11
  17. 2022Unsupervised Point Cloud Pre-Training via Contrasting and Clusteringcited by 9
  18. 2025PedDet: Adaptive Spectral Optimization for Multimodal Pedestrian Detectioncited by 6
  19. 2024Bi-directional Contextual Attention for 3D Dense Captioningcited by 5
  20. 2024See It All: Contextualized Late Aggregation for 3D Dense Captioningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 6.

13 papers.

Sort:

By area:

Similar, not cited

Close in wording, no citation either way

  1. 2023Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes
  2. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning
  3. 2024Grounded 3D-LLM with Referent Tokens
  4. 20243D Scene Graph Guided Vision-Language Pre-training
  5. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes
  6. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data
  7. 2025UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding
  8. 2023Lang3DSG: Language-based contrastive pre-training for 3D Scene Graph prediction