mlsanityData to 14 Oct 2026

Point Clouds Are Specialized Images: A Knowledge Transfer Approach for 3D Understanding

arXiv 2307.15569 ↗First posted 28 Jul 2023cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Self-supervised representation learning (SSRL) has gained increasing attention in point cloud understanding, in addressing the challenges posed by 3D data scarcity and high annotation costs. This paper presents PCExpert, a novel SSRL approach that reinterprets point clouds as "specialized images". This conceptual shift allows PCExpert to leverage knowledge derived from large-scale image modality in a more direct and deeper manner, via extensively sharing the parameters with a pre-trained image encoder in a multi-way Transformer architecture. The parameter sharing strategy, combined with a novel pretext task for pre-training, i.e., transformation estimation, empowers PCExpert to outperform the state of the arts in a variety of tasks, with a remarkable reduction in the number of trainable parameters. Notably, PCExpert's performance under LINEAR fine-tuning (e.g., yielding a 90.02% overall accuracy on ScanObjectNN) has already approached the results obtained with FULL model fine-tuning (92.66%), demonstrating its effective and robust representation capability.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 39 of its 58 references that are in the map. First posted on arXiv 28 Jul 2023.

Builds on

39 of 58 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  10. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
  11. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  12. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 784
27 more
  1. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  2. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  3. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  4. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  5. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  6. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 436
  7. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  8. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 379
  9. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  10. 2019DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstructioncited by 333
  11. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 267
  12. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 264
  13. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  14. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  15. 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
  16. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 164
  17. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 122
  18. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  19. 2022P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Promptingcited by 74
  20. 2021SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representationscited by 55
  21. 2021Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Modelscited by 41
  22. 2022LidarCLIP or: How I Learned to Talk to Point Cloudscited by 33
  23. 2022A Closer Look at Invariances in Self-supervised Pre-training for 3D Visioncited by 24
  24. 2022Self-Supervised Learning with Multi-View Rendering for 3D Point Cloud Analysiscited by 8
  25. 2023ViPFormer: Efficient Vision-and-Pointcloud Transformer for Unsupervised Pointcloud Understandingcited by 8
  26. 2022PointCMC: Cross-Modal Multi-Scale Correspondences Learning for Point Cloud Understandingcited by 5
  27. 2022Leveraging Systematic Knowledge of 2D Transformationscited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2024Positional Prompt Tuning for Efficient 3D Representation Learningcited by 15
  2. 2023Language-Assisted 3D Scene Understandingcited by 6
  3. 2024Boosting Cross-Domain Point Classification via Distilling Relational Priors from 2D Transformerscited by 0