mlsanityData to 7 Oct 2026

3D Point Cloud Pre-training with Knowledge Distillation from 2D Images

arXiv 2212.08974 ↗First posted 17 Dec 2022cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

The recent success of pre-trained 2D vision models is mostly attributable to learning from large-scale datasets. However, compared with 2D image datasets, the current pre-training data of 3D point cloud is limited. To overcome this limitation, we propose a knowledge distillation method for 3D point cloud pre-trained models to acquire knowledge directly from the 2D representation learning model, particularly the image encoder of CLIP, through concept alignment. Specifically, we introduce a cross-attention mechanism to extract concept features from 3D point cloud and compare them with the semantic information from 2D images. In this scheme, the point cloud pre-trained models learn directly from rich information contained in 2D teacher models. Extensive experiments demonstrate that the proposed knowledge distillation scheme achieves higher accuracy than the state-of-the-art 3D pre-training methods for synthetic and real-world datasets on downstream tasks, including object classification, object detection, semantic segmentation, and part segmentation.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 44 of its 51 references that are in the map. First posted on arXiv 17 Dec 2022.

Builds on

44 of 51 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  7. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  10. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  12. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
32 more
  1. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  2. 2020Knowledge Distillation: A Surveycited by 1,578
  3. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  4. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  5. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  6. 2019Contrastive Representation Distillationcited by 734
  7. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  8. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  9. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  10. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  11. 2019Similarity-Preserving Knowledge Distillationcited by 507
  12. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  13. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  14. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  15. 2019Relation-Shape Convolutional Neural Network for Point Cloud Analysiscited by 346
  16. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  17. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  18. 2019Correlation Congruence for Knowledge Distillationcited by 280
  19. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  20. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  21. 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
  22. 2021Spatio-temporal Self-Supervised Representation Learning for 3D Point Cloudscited by 180
  23. 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155
  24. 2020Cross-Layer Distillation with Semantic Calibrationcited by 141
  25. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  26. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  27. 2019Knowledge Distillation via Route Constrained Optimizationcited by 73
  28. 2020Self-Supervised Few-Shot Learning on Point Cloudscited by 66
  29. 2018Self-supervised Knowledge Distillation Using Singular Value Decompositioncited by 47
  30. 2022Implicit Autoencoder for Point-Cloud Self-Supervised Representation Learningcited by 46
  31. 2021RandomRooms: Unsupervised Pre-training from Synthetic Shapes and Randomized Layouts for 3D Object Detectioncited by 37
  32. 2022Visual Concepts Tokenizationcited by 14

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 3; 2025: 1.

7 papers.

  1. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  2. 2025Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understandingcited by 13
  3. 2024Harnessing Uncertainty-aware Bounding Boxes for Unsupervised 3D Object Detectioncited by 9
  4. 2023Language-Assisted 3D Scene Understandingcited by 6
  5. 2024Vision-Language Guidance for LiDAR-based Unsupervised 3D Object Detectioncited by 3
  6. 2024Adaptive Point Transformercited by 1
  7. 2023Smaller3d: Smaller Models for 3D Semantic Segmentation Using Minkowski Engine and Knowledge Distillation Methodscited by 0