mlsanityData to 14 Oct 2026

CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-training

arXiv 2210.01055 ↗First posted 3 Oct 2022cs.CVcited by 160

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pre-training across 3D vision and language remains under development because of limited training data. Recent works attempt to transfer vision-language pre-training models to 3D vision. PointCLIP converts point cloud data to multi-view depth maps, adopting CLIP for shape classification. However, its performance is restricted by the domain gap between rendered depth maps and images, as well as the diversity of depth distributions. To address this issue, we propose CLIP2Point, an image-depth pre-training method by contrastive learning to transfer CLIP to the 3D domain, and adapt it to point cloud classification. We introduce a new depth rendering setting that forms a better visual effect, and then render 52,460 pairs of images and depth maps from ShapeNet for pre-training. The pre-training scheme of CLIP2Point combines cross-modality learning to enforce the depth features for capturing expressive visual and textual features and intra-modality learning to enhance the invariance of depth aggregation. Additionally, we propose a novel Dual-Path Adapter (DPA) module, i.e., a dual-path structure with simplified adapters for few-shot learning. The dual-path structure allows the joint use of CLIP and CLIP2Point, and the simplified adapter can well fit few-shot tasks without post-search. Experimental results show that CLIP2Point is effective in transferring CLIP knowledge to 3D vision. Our CLIP2Point outperforms PointCLIP and other self-supervised 3D networks, achieving state-of-the-art results on zero-shot and few-shot classification.

Key facts. Cited by 160 papers in the map (data through 14 Oct 2026). Builds on 48 of its 62 references that are in the map. First posted on arXiv 3 Oct 2022.

Builds on

48 of 62 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
36 more
  1. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  3. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
  4. 2022Visual Prompt Tuningcited by 1,681
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  6. 2020Point Transformercited by 1,470
  7. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  8. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  9. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  10. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  11. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  12. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  13. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 588
  14. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 565
  15. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  16. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 504
  17. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  18. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  19. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  20. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  21. 2019DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstructioncited by 333
  22. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
  23. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  24. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  25. 2021Revisiting Point Cloud Shape Classification with a Simple and Effective Baselinecited by 145
  26. 2020MVTN: Multi-View Transformation Network for 3D Shape Recognitioncited by 140
  27. 2018Gated Fusion Network for Single Image Dehazingcited by 133
  28. 2022P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Promptingcited by 74
  29. 2022Towards Implicit Text-Guided 3D Shape Generationcited by 68
  30. 2021Zero-Shot Learning on 3D Point Cloud Objects and Beyondcited by 38
  31. 2019Zero-shot Learning of 3D Point Cloud Objectscited by 34
  32. 2018Y^2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequencescited by 27
  33. 2022A Closer Look at Invariances in Self-supervised Pre-training for 3D Visioncited by 24
  34. 2022Open-Vocabulary 3D Detection via Image-level Class and Debiased Cross-modal Contrastive Learningcited by 23
  35. 2016Orientation-boosted Voxel Nets for 3D Object Recognitioncited by 17
  36. 2022Unsupervised Point Cloud Pre-Training via Contrasting and Clusteringcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 39; 2024: 54; 2025: 40; 2026: 22.

160 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  3. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  4. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 237
  5. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 208
  6. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 202
  7. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  8. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 187
  9. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 184
  10. 2023Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detectioncited by 182
  11. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  12. 2023Towards Open Vocabulary Learning: A Surveycited by 166
38 more
  1. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 164
  2. 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 145
  3. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  4. 2023Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understandingcited by 113
  5. 2024C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detectioncited by 113
  6. 2024SAMPart3D: Segment Any Part in 3D Objectscited by 97
  7. 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 94
  8. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  9. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  10. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 76
  11. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 71
  12. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  13. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69
  14. 2023Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Modelscited by 68
  15. 2023Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labelscited by 64
  16. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
  17. 2023Point Cloud Pre-training with Diffusion Modelscited by 47
  18. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  19. 2024DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priorscited by 41
  20. 2023DreamControl: Control-Based Text-to-3D Generation with 3D Self-Priorcited by 39
  21. 2023Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understandingcited by 38
  22. 2024SGD: Street View Synthesis with Gaussian Splatting and Diffusion Priorcited by 36
  23. 2024InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perceptioncited by 35
  24. 2023Parameter is Not All You Need: Starting from Non-Parametric Networks for 3D Point Cloud Analysiscited by 34
  25. 2022LidarCLIP or: How I Learned to Talk to Point Cloudscited by 33
  26. 2024Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalitiescited by 31
  27. 2024TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understandingcited by 27
  28. 2024SUGAR: Pre-training 3D Visual Representations for Roboticscited by 26
  29. 2023Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understandingcited by 25
  30. 2023Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detectioncited by 24
  31. 2023DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classificationcited by 23
  32. 2024Securely Fine-tuning Pre-trained Encoders Against Adversarial Examplescited by 21
  33. 2024Find n' Propagate: Open-Vocabulary 3D Object Detection in Urban Environmentscited by 21
  34. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  35. 2023A Survey of Label-Efficient Deep Learning for 3D Point Cloudscited by 20
  36. 2024Rethinking Prior Information Generation with CLIP for Few-Shot Segmentationcited by 20
  37. 2024OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of Allcited by 20
  38. 2025RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learningcited by 20

50 most cited of 160 All 160 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Joint Representation Learning for Text and 3D Point Cloud
  2. 2025Parameter-Efficient CLIP Adaptation for 3D Understanding via Unified Tokenization
  3. 2023Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes
  4. 20223D Point Cloud Pre-training with Knowledge Distillation from 2D Images
  5. 2022Let Images Give You More:Point Cloud Cross-Modal Training for Shape Analysis
  6. 2026Lightweight Prompt-Guided CLIP Adaptation for Monocular Depth Estimation
  7. 2022Self-Supervised Learning with Multi-View Rendering for 3D Point Cloud Analysis
  8. 2023CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement