mlsanityData to 14 Oct 2026

PointCLIP: Point Cloud Understanding by CLIP

arXiv 2112.02413 ↗First posted 4 Dec 2021cs.CV cs.AI cs.ROcited by 463

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recently, zero-shot and few-shot learning via Contrastive Vision-Language Pre-training (CLIP) have shown inspirational performance on 2D visual recognition, which learns to match images with their corresponding texts in open-vocabulary settings. However, it remains under explored that whether CLIP, pre-trained by large-scale image-text pairs in 2D, can be generalized to 3D recognition. In this paper, we identify such a setting is feasible by proposing PointCLIP, which conducts alignment between CLIP-encoded point cloud and 3D category texts. Specifically, we encode a point cloud by projecting it into multi-view depth maps without rendering, and aggregate the view-wise zero-shot prediction to achieve knowledge transfer from 2D to 3D. On top of that, we design an inter-view adapter to better extract the global feature and adaptively fuse the few-shot knowledge learned from 3D into CLIP pre-trained in 2D. By just fine-tuning the lightweight adapter in the few-shot settings, the performance of PointCLIP could be largely improved. In addition, we observe the complementary property between PointCLIP and classical 3D-supervised networks. By simple ensembling, PointCLIP boosts baseline's performance and even surpasses state-of-the-art models. Therefore, PointCLIP is a promising alternative for effective 3D point cloud understanding via CLIP under low resource cost and data regime. We conduct thorough experiments on widely-adopted ModelNet10, ModelNet40 and the challenging ScanObjectNN to demonstrate the effectiveness of PointCLIP. The code is released at https://github.com/ZrrSkywalker/PointCLIP.

Key facts. Cited by 463 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 71 references that are in the map. First posted on arXiv 4 Dec 2021.

Builds on

60 of 71 references.

References · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  9. 2020End-to-End Object Detection with Transformerscited by 7,414
  10. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  11. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  12. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  3. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  4. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  5. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
  6. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  7. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  8. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,205
  9. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  10. 2021Robust fine-tuning of zero-shot modelscited by 797
  11. 2018Image Transformercited by 772
  12. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  13. 2020PCT: Point cloud transformercited by 732
  14. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 724
  15. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 718
  16. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  17. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  18. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 565
  19. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  20. 2018Hypergraph Neural Networkscited by 488
  21. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  22. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  23. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  24. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 366
  25. 2019Relation-Shape Convolutional Neural Network for Point Cloud Analysiscited by 346
  26. 2019Point-Voxel CNN for Efficient 3D Deep Learningcited by 343
  27. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  28. 2018Context Encoding for Semantic Segmentationcited by 290
  29. 2021Fast Convergence of DETR with Spatially Modulated Co-Attentioncited by 176
  30. 2021Walk in the Cloud: Learning Curves for Point Clouds Shape Analysiscited by 172
  31. 2016Learning a Deep Embedding Model for Zero-Shot Learningcited by 153
  32. 2021Revisiting Point Cloud Shape Classification with a Simple and Effective Baselinecited by 145
  33. 2020MVTN: Multi-View Transformation Network for 3D Shape Recognitioncited by 140
  34. 2016Latent Embeddings for Zero-shot Classificationcited by 129
  35. 2019MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequencescited by 117
  36. 2020End-to-End Object Detection with Adaptive Clustering Transformercited by 109
  37. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  38. 2016Vote3Deep: Fast Object Detection in 3D Point Clouds Using Efficient Convolutional Neural Networkscited by 72

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 37; 2023: 131; 2024: 128; 2025: 102; 2026: 65.

463 papers.

Sort:

By area:

  1. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  2. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  3. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  4. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  5. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  6. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  7. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  8. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  9. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 362
  10. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  11. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  12. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 267
38 more
  1. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 264
  2. 2023Personalize Segment Anything Model with One Shotcited by 250
  3. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  4. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 237
  5. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 228
  6. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 208
  7. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 202
  8. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  9. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 187
  10. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  11. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 184
  12. 2022Frozen CLIP Models are Efficient Video Learnerscited by 182
  13. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 181
  14. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  15. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  16. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 164
  17. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 160
  18. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  19. 2022Unified Vision and Language Prompt Learningcited by 148
  20. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  21. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 144
  22. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  23. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  24. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  25. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  26. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 127
  27. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 122
  28. 2022PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Modelscited by 118
  29. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  30. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  31. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 103
  32. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 99
  33. 2024SAMPart3D: Segment Any Part in 3D Objectscited by 97
  34. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 95
  35. 2025PARTFIELD: Learning 3D Feature Fields for Part Segmentation and Beyondcited by 95
  36. 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 94
  37. 2023Vita-CLIP: Video and text adaptive CLIP via Multimodal Promptingcited by 91
  38. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89

50 most cited of 463 All 463 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024UniPLV: Towards Label-Efficient Open-World 3D Scene Understanding by Regional Visual Language Supervision
  2. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes
  3. 2024More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding
  4. 2024ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models
  5. 20223D Point Cloud Pre-training with Knowledge Distillation from 2D Images
  6. 2024Cross-Modal Self-Training: Aligning Images and Pointclouds to Learn Classification without Labels
  7. 2023Language-Assisted 3D Scene Understanding
  8. 2026PointLLM-R: Enhancing 3D Point Cloud Reasoning via Chain-of-Thought