mlsanityData to 7 Oct 2026

Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning

arXiv 2202.04241 ↗First posted 9 Feb 2022cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a simple and general framework for self-supervised point cloud representation learning. Human beings understand the 3D world by extracting two levels of information and establishing the relationship between them. One is the global shape of an object, and the other is the local structures of it. However, few existing studies in point cloud representation learning explored how to learn both global shapes and local-to-global relationships without a specified network architecture. Inspired by how human beings understand the world, we utilize knowledge distillation to learn both global shape information and the relationship between global shape and local structures. At the same time, we combine contrastive learning with knowledge distillation to make the teacher network be better updated. Our method achieves the state-of-the-art performance on linear classification and multiple other downstream tasks. Especially, we develop a variant of ViT for 3D point cloud feature extraction, which also achieves comparable results with existing backbones when combined with our framework, and visualization of the attention maps show that our model does understand the point cloud by combining the global shape information and multiple local structural information, which is consistent with the inspiration of our representation learning method. Our code will be released soon.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 9 Feb 2022.

Builds on

38 of 51 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  10. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  12. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
26 more
  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  2. 2017Decoupled Weight Decay Regularizationcited by 1,722
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  4. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  5. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  6. 2020PCT: Point cloud transformercited by 730
  7. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  8. 2013Do Deep Nets Really Need to be Deep?cited by 466
  9. 2019Variational Information Distillation for Knowledge Transfercited by 321
  10. 2020PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Samplingcited by 238
  11. 2016Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clusteringcited by 237
  12. 2018SO-Net: Self-Organizing Network for Point Cloud Analysiscited by 210
  13. 2021Spatio-temporal Self-Supervised Representation Learning for 3D Point Cloudscited by 180
  14. 2021Fast Convergence of DETR with Spatially Modulated Co-Attentioncited by 176
  15. 2021Walk in the Cloud: Learning Curves for Point Clouds Shape Analysiscited by 172
  16. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  17. 2020Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Reviewcited by 107
  18. 20183D Point Capsule Networkscited by 106
  19. 2019ShellNet: Efficient Point Cloud Convolutional Neural Networks using Concentric Shells Statisticscited by 104
  20. 2018Multiresolution Tree Networks for 3D Point Cloud Processingcited by 91
  21. 2020Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Cloudscited by 75
  22. 2021Self Supervision to Distillation for Long-Tailed Visual Recognitioncited by 73
  23. 2020Self-supervised Learning of Point Clouds via Orientation Estimationcited by 72
  24. 2019Multi-Angle Point Cloud-VAE: Unsupervised Feature Learning for 3D Point Clouds from Multiple Angles by Joint Self-Reconstruction and Half-to-Half Predictioncited by 66
  25. 2018View Inter-Prediction GAN: Unsupervised Representation Learning for 3D Shapes by Learning Global Shape Memories to Support Local View Predictionscited by 43
  26. 2021Container: Context Aggregation Networkcited by 37

Built on by

Cited by year

Citing papers per year: 2022: 7; 2023: 8; 2024: 3; 2025: 1.

19 papers.

Sort:

  1. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  2. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  3. 2023Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-trainingcited by 52
  4. 2022TiG-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learningcited by 37
  5. 2023Parameter is Not All You Need: Starting from Non-Parametric Networks for 3D Point Cloud Analysiscited by 34
  6. 2022Transformers in 3D Point Clouds: A Surveycited by 31
  7. 2022Complete-to-Partial 4D Distillation for Self-Supervised Point Cloud Sequence Representation Learningcited by 30
  8. 2024Point-JEPA: A Joint Embedding Predictive Architecture for Self-Supervised Learning on Point Cloudcited by 29
  9. 20223DCTN: 3D Convolution-Transformer Network for Point Cloud Classificationcited by 25
  10. 2023Self-supervised Learning for Pre-Training 3D Point Clouds: A Surveycited by 21
  11. 2022Rethinking the compositionality of point clouds through regularization in the hyperbolic spacecited by 18
  12. 20243D-JEPA: A Joint Embedding Predictive Architecture for 3D Self-Supervised Representation Learningcited by 14
7 more
  1. 2023Regress Before Construct: Regress Autoencoder for Point Cloud Self-supervised Learningcited by 6
  2. 2024TiGDistill-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learning Distillationcited by 4
  3. 2022SeRP: Self-Supervised Representation Learning Using Perturbed Point Cloudscited by 3
  4. 2023PointSmile: Point Self-supervised Learning via Curriculum Mutual Informationcited by 2
  5. 2023Self-Ordering Point Cloudscited by 2
  6. 2023Self-supervised Learning of Rotation-invariant 3D Point Set Features using Transformer and its Self-distillationcited by 2
  7. 2025Joint graph entropy knowledge distillation for point cloud classification and robustness against corruptionscited by 0