3D CoCa: Contrastive Learners are 3D Captioners
Not on the map. Why?
Abstract (arXiv)
3D captioning, which aims to describe the content of 3D scenes in natural language, remains highly challenging due to the inherent sparsity of point clouds and weak cross-modal alignment in existing methods. To address these challenges, we propose 3D CoCa, a novel unified framework that seamlessly combines contrastive vision-language learning with 3D caption generation in a single architecture. Our approach leverages a frozen CLIP vision-language backbone to provide rich semantic priors, a spatially-aware 3D scene encoder to capture geometric context, and a multi-modal decoder to generate descriptive captions. Unlike prior two-stage methods that rely on explicit object proposals, 3D CoCa jointly optimizes contrastive and captioning objectives in a shared feature space, eliminating the need for external detectors or handcrafted proposals. This joint training paradigm yields stronger spatial reasoning and richer semantic grounding by aligning 3D and textual representations. Extensive experiments on the ScanRefer and Nr3D benchmarks demonstrate that 3D CoCa significantly outperforms current state-of-the-arts by 10.2% and 5.76% in CIDEr at 0.5IoU, respectively. Code will be available at https://github.com/AIGeeksGroup/3DCoCa.
Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 13 Apr 2025.
Builds on
32 of 49 references.
References · 32
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
- 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
- 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
- 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
- 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 504
- 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
- 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
- 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
- 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
- 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
20 more
- 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
- 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 78
- 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
- 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
- 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
- 2021SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videoscited by 40
- 2022Spatiality-guided Transformer for 3D Dense Captioning on Point Cloudscited by 40
- 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
- 2023Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioningcited by 39
- 2019TightCap: 3D Human Shape Capture with Clothing Tightness Fieldcited by 25
- 2023Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understandingcited by 25
- 2023Take-A-Photo: 3D-to-2D Generative Pre-training of Point Cloud Modelscited by 25
- 2022EPCL: Frozen CLIP Transformer is An Efficient Point Cloud Encodercited by 18
- 2021Point Cloud Instance Segmentation with Semi-supervised Bounding-Box Miningcited by 15
- 2024MSDet: Receptive Field Enhanced Multiscale Detection for Tiny Pulmonary Nodulecited by 14
- 2024MedDet: Generative Adversarial Distillation for Efficient Cervical Disc Herniation Detectioncited by 11
- 2022Unsupervised Point Cloud Pre-Training via Contrasting and Clusteringcited by 9
- 2025PedDet: Adaptive Spectral Optimization for Multimodal Pedestrian Detectioncited by 6
- 2024Bi-directional Contextual Attention for 3D Dense Captioningcited by 5
- 2024See It All: Contextualized Late Aggregation for 3D Dense Captioningcited by 4
Built on by
Citing papers per year: 2025: 7; 2026: 6.
13 papers.
Sort:
By area:
- 20253D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understandingcited by 47
- 2025VLA-R1: Enhancing Reasoning in Vision-Language-Action Modelscited by 38
- 2025DC-Scene: Data-Centric Learning for 3D Scene Understandingcited by 15
- 2025MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robotscited by 15
- 2025EvoVLA: Self-Evolving Vision-Language-Action Modelcited by 11
- 2025StereoAdapter: Adapting Stereo Depth Estimation to Underwater Scenescited by 9
- 2026CoV: Chain-of-View Prompting for Spatial Reasoningcited by 9
- 2026AnyDepth: Depth Estimation Made Easycited by 5
- 2025VaseVQA-3D: Benchmarking 3D VLMs on Ancient Greek Potterycited by 4
- 20263D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligencecited by 3
- 2026UniMesh: Unifying 3D Mesh Understanding and Generationcited by 2
- 2026ConsiSpace: Learning Geometric Consistency Matters for Video Spatial Reasoningcited by 1
1 more
- 2026MobileVLA-R1 2.0: RL-Enhanced Reasoning for Mobile Robot Control5 Sep 2026
- 2026ConsiSpace: Learning Geometric Consistency Matters for Video Spatial Reasoning20 Jul 2026
- 2026UniMesh: Unifying 3D Mesh Understanding and Generation19 Apr 2026
- 20263D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligence10 Jan 2026
- 2026CoV: Chain-of-View Prompting for Spatial Reasoning8 Jan 2026
- 2026AnyDepth: Depth Estimation Made Easy6 Jan 2026
- 2025MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robots22 Nov 2025
- 2025EvoVLA: Self-Evolving Vision-Language-Action Model20 Nov 2025
- 2025VaseVQA-3D: Benchmarking 3D VLMs on Ancient Greek Pottery6 Oct 2025
- 2025VLA-R1: Enhancing Reasoning in Vision-Language-Action Models2 Oct 2025
- 2025StereoAdapter: Adapting Stereo Depth Estimation to Underwater Scenes19 Sep 2025
- 20253D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding31 Jul 2025
1 more
- 2025DC-Scene: Data-Centric Learning for 3D Scene Understanding21 May 2025
- 20253D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understandingcited by 47
- 2026CoV: Chain-of-View Prompting for Spatial Reasoningcited by 9
- 2025VaseVQA-3D: Benchmarking 3D VLMs on Ancient Greek Potterycited by 4
- 2026ConsiSpace: Learning Geometric Consistency Matters for Video Spatial Reasoningcited by 1
- 2026ConsiSpace: Learning Geometric Consistency Matters for Video Spatial Reasoning20 Jul 2026
- 2026CoV: Chain-of-View Prompting for Spatial Reasoning8 Jan 2026
- 2025VaseVQA-3D: Benchmarking 3D VLMs on Ancient Greek Pottery6 Oct 2025
- 20253D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding31 Jul 2025
- 2025DC-Scene: Data-Centric Learning for 3D Scene Understandingcited by 15
- 2025StereoAdapter: Adapting Stereo Depth Estimation to Underwater Scenescited by 9
- 2026AnyDepth: Depth Estimation Made Easycited by 5
- 20263D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligencecited by 3
- 2026UniMesh: Unifying 3D Mesh Understanding and Generationcited by 2
- 2026UniMesh: Unifying 3D Mesh Understanding and Generation19 Apr 2026
- 20263D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligence10 Jan 2026
- 2026AnyDepth: Depth Estimation Made Easy6 Jan 2026
- 2025StereoAdapter: Adapting Stereo Depth Estimation to Underwater Scenes19 Sep 2025
- 2025DC-Scene: Data-Centric Learning for 3D Scene Understanding21 May 2025
Similar, not cited
Close in wording, no citation either way
- 2023Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes
- 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning
- 2024Grounded 3D-LLM with Referent Tokens
- 20243D Scene Graph Guided Vision-Language Pre-training
- 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes
- 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data
- 2025UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding
- 2023Lang3DSG: Language-based contrastive pre-training for 3D Scene Graph prediction