mlsanityData to 7 Oct 2026

Data Efficient 3D Learner via Knowledge Transferred from 2D Model

arXiv 2203.08479 ↗First posted 16 Mar 2022cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Collecting and labeling the registered 3D point cloud is costly. As a result, 3D resources for training are typically limited in quantity compared to the 2D images counterpart. In this work, we deal with the data scarcity challenge of 3D tasks by transferring knowledge from strong 2D models via RGB-D images. Specifically, we utilize a strong and well-trained semantic segmentation model for 2D images to augment RGB-D images with pseudo-label. The augmented dataset can then be used to pre-train 3D models. Finally, by simply fine-tuning on a few labeled 3D instances, our method already outperforms existing state-of-the-art that is tailored for 3D label efficiency. We also show that the results of mean-teacher and entropy minimization can be improved by our pre-training, suggesting that the transferred knowledge is helpful in semi-supervised setting. We verify the effectiveness of our approach on two popular 3D models and three different tasks. On ScanNet official evaluation, we establish new state-of-the-art semantic segmentation results on the data-efficient track.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 39 of its 49 references that are in the map. First posted on arXiv 16 Mar 2022.

Builds on

39 of 49 references.

References · 39

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  10. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  12. 2020Exploring Simple Siamese Representation Learningcited by 2,853
27 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  2. 2016Pyramid Scene Parsing Networkcited by 2,472
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  4. 2021Vision Transformers for Dense Predictioncited by 1,789
  5. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  6. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  7. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  8. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  9. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  10. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  11. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
  12. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 573
  13. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  14. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 340
  15. 2020PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Samplingcited by 238
  16. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  17. 2020Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contextscited by 210
  18. 2019Asymmetric Non-local Neural Networks for Semantic Segmentationcited by 196
  19. 20183D Point Capsule Networkscited by 106
  20. 2019Unsupervised Multi-Task Feature Learning on Point Cloudscited by 89
  21. 2021One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentationcited by 89
  22. 2015Sparse 3D convolutional neural networkscited by 71
  23. 2019ConvPoint: Continuous Convolutions for Point Cloud Processingcited by 62
  24. 2021Learning 3D Semantic Segmentation with only 2D Image Supervisioncited by 60
  25. 2020Unsupervised 3D Learning for Shape Analysis via Multiresolution Instance Discriminationcited by 31
  26. 2021Specialize and Fuse: Pyramidal Output Representation for Semantic Segmentationcited by 7
  27. 2021Pointly-supervised 3D Scene Parsing with Viewpoint Bottleneckcited by 7

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 2; 2025: 1.

10 papers.

  1. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  2. 2023Three Pillars improving Vision Foundation Model Distillation for Lidarcited by 43
  3. 2023PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillationcited by 27
  4. 2023Hierarchical Point-based Active Learning for Semi-supervised Point Cloud Semantic Segmentationcited by 21
  5. 2024ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Imagescited by 17
  6. 20232D-3D Interlaced Transformer for Point Cloud Segmentation with Scene-Level Supervisioncited by 16
  7. 2024Advancing 3D Point Cloud Understanding through Deep Transfer Learning: A Comprehensive Surveycited by 9
  8. 2023Generalized Robot 3D Vision-Language Model with Fast Rendering and Pre-Training Vision-Language Alignmentcited by 1
  9. 2023You Only Need One Thing One Click: Self-Training for Weakly Supervised 3D Scene Understandingcited by 0
  10. 2025Adaptive Point-Prompt Tuning: Fine-Tuning Heterogeneous Foundation Models for 3D Point Cloud Analysiscited by 0