# 3D CoCa: Contrastive Learners are 3D Captioners arXiv: https://arxiv.org/abs/2504.09518 ยท HTML: https://mlsanity.com/paper/2504.09518/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 32 of its 49 references that are in the map. - First posted on arXiv 13 Apr 2025. ## Builds on (most cited first, 30 of 32) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017) - [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017) - [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014) - [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022) - [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021) - [PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding](https://mlsanity.com/paper/2007.10985/) (2020) - [Masked Autoencoders for Point Cloud Self-supervised Learning](https://mlsanity.com/paper/2203.06604/) (2022) - [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019) - [Scan2Cap: Context-aware Dense Captioning in RGB-D Scans](https://mlsanity.com/paper/2012.02206/) (2020) - [LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning](https://mlsanity.com/paper/2311.18651/) (2023) - [Unsupervised Point Cloud Pre-Training via Occlusion Completion](https://mlsanity.com/paper/2010.01089/) (2020) - [Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers](https://mlsanity.com/paper/2312.08168/) (2023) - [End-to-End 3D Dense Captioning with Vote2Cap-DETR](https://mlsanity.com/paper/2301.02508/) (2023) - [X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning](https://mlsanity.com/paper/2203.00843/) (2022) - [UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding](https://mlsanity.com/paper/2212.00836/) (2022) - [MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenes](https://mlsanity.com/paper/2203.05203/) (2022) - [SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos](https://mlsanity.com/paper/2104.11452/) (2021) - [Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds](https://mlsanity.com/paper/2204.10688/) (2022) - [TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenes](https://mlsanity.com/paper/2403.19589/) (2024) - [Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioning](https://mlsanity.com/paper/2309.02999/) (2023) - [TightCap: 3D Human Shape Capture with Clothing Tightness Field](https://mlsanity.com/paper/1904.02601/) (2019) - [Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding](https://mlsanity.com/paper/2305.10714/) (2023) - [Take-A-Photo: 3D-to-2D Generative Pre-training of Point Cloud Models](https://mlsanity.com/paper/2307.14971/) (2023) - [EPCL: Frozen CLIP Transformer is An Efficient Point Cloud Encoder](https://mlsanity.com/paper/2212.04098/) (2022) - [Point Cloud Instance Segmentation with Semi-supervised Bounding-Box Mining](https://mlsanity.com/paper/2111.15210/) (2021) - [MSDet: Receptive Field Enhanced Multiscale Detection for Tiny Pulmonary Nodule](https://mlsanity.com/paper/2409.14028/) (2024) - [MedDet: Generative Adversarial Distillation for Efficient Cervical Disc Herniation Detection](https://mlsanity.com/paper/2409.00204/) (2024) - [Unsupervised Point Cloud Pre-Training via Contrasting and Clustering](https://mlsanity.com/paper/2202.02543/) (2022) - [PedDet: Adaptive Spectral Optimization for Multimodal Pedestrian Detection](https://mlsanity.com/paper/2502.14063/) (2025) ## Built on by (most cited first, 13 of 13) - [3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding](https://mlsanity.com/paper/2507.23478/) (2025) - [VLA-R1: Enhancing Reasoning in Vision-Language-Action Models](https://mlsanity.com/paper/2510.01623/) (2025) - [DC-Scene: Data-Centric Learning for 3D Scene Understanding](https://mlsanity.com/paper/2505.15232/) (2025) - [MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robots](https://mlsanity.com/paper/2511.17889/) (2025) - [EvoVLA: Self-Evolving Vision-Language-Action Model](https://mlsanity.com/paper/2511.16166/) (2025) - [StereoAdapter: Adapting Stereo Depth Estimation to Underwater Scenes](https://mlsanity.com/paper/2509.16415/) (2025) - [CoV: Chain-of-View Prompting for Spatial Reasoning](https://mlsanity.com/paper/2601.05172/) (2026) - [AnyDepth: Depth Estimation Made Easy](https://mlsanity.com/paper/2601.02760/) (2026) - [VaseVQA-3D: Benchmarking 3D VLMs on Ancient Greek Pottery](https://mlsanity.com/paper/2510.04479/) (2025) - [3D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligence](https://mlsanity.com/paper/2601.06496/) (2026) - [UniMesh: Unifying 3D Mesh Understanding and Generation](https://mlsanity.com/paper/2604.17472/) (2026) - [ConsiSpace: Learning Geometric Consistency Matters for Video Spatial Reasoning](https://mlsanity.com/paper/2607.17599/) (2026) - [MobileVLA-R1 2.0: RL-Enhanced Reasoning for Mobile Robot Control](https://mlsanity.com/paper/2609.06251/) (2026) ## Abstract (arXiv) 3D captioning, which aims to describe the content of 3D scenes in natural language, remains highly challenging due to the inherent sparsity of point clouds and weak cross-modal alignment in existing methods. To address these challenges, we propose 3D CoCa, a novel unified framework that seamlessly combines contrastive vision-language learning with 3D caption generation in a single architecture. Our approach leverages a frozen CLIP vision-language backbone to provide rich semantic priors, a spatially-aware 3D scene encoder to capture geometric context, and a multi-modal decoder to generate descriptive captions. Unlike prior two-stage methods that rely on explicit object proposals, 3D CoCa jointly optimizes contrastive and captioning objectives in a shared feature space, eliminating the need for external detectors or handcrafted proposals. This joint training paradigm yields stronger spatial reasoning and richer semantic grounding by aligning 3D and textual representations. Extensive experiments on the ScanRefer and Nr3D benchmarks demonstrate that 3D CoCa significantly outperforms current state-of-the-arts by 10.2% and 5.76% in CIDEr at 0.5IoU, respectively. Code will be available at https://github.com/AIGeeksGroup/3DCoCa.