mlsanityData to 7 Oct 2026

Self-Supervised Pretraining of 3D Features on any Point-Cloud

arXiv 2101.02691 ↗First posted 7 Jan 2021cs.CVcited by 217

Not on the map. Why?

Abstract (arXiv)

Pretraining on large labeled datasets is a prerequisite to achieve good performance in many computer vision tasks like 2D object recognition, video classification etc. However, pretraining is not widely used for 3D recognition tasks where state-of-the-art methods train models from scratch. A primary reason is the lack of large annotated datasets because 3D data is both difficult to acquire and time consuming to label. We present a simple self-supervised pertaining method that can work with any 3D data - single or multiview, indoor or outdoor, acquired by varied sensors, without 3D registration. We pretrain standard point cloud and voxel based model architectures, and show that joint pretraining further improves performance. We evaluate our models on 9 benchmarks for object detection, semantic segmentation, and object classification, where they achieve state-of-the-art results and can outperform supervised pretraining. We set a new state-of-the-art for object detection on ScanNet (69.0% mAP) and SUNRGBD (63.5% mAP). Our pretrained models are label efficient and improve performance for classes with few examples.

Key facts. Cited by 217 papers in the map (data through 7 Oct 2026). Builds on 85 of its 120 references that are in the map. First posted on arXiv 7 Jan 2021.

Builds on

85 of 120 references.

References · 85

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  8. 2014Going Deeper with Convolutionscited by 4,231
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  10. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
38 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  2. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  3. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  4. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  5. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  6. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  7. 2019Contrastive Multiview Codingcited by 1,297
  8. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  9. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  10. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  11. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  12. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  13. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  14. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 890
  15. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  16. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  17. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  18. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  19. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  20. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  21. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
  22. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  23. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  24. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  25. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  26. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 564
  27. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  28. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  29. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 454
  30. 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 404
  31. 2019From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Networkcited by 366
  32. 2016Adversarial Feature Learningcited by 350
  33. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 340
  34. 2019Deep Closest Point: Learning Representations for Point Cloud Registrationcited by 318
  35. 2019STD: Sparse-to-Dense 3D Object Detector for Point Cloudcited by 282
  36. 2018SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filterscited by 261
  37. 2019Revisiting Self-Supervised Visual Representation Learningcited by 250
  38. 2017Learning SO(3) Equivariant Representations with Spherical CNNscited by 244

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 47; 2023: 72; 2024: 43; 2025: 31; 2026: 11.

217 papers.

Sort:

By area:

  1. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  2. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  3. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 376
  4. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  5. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  6. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  7. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  8. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  9. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  10. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  11. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
  12. 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155
38 more
  1. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  2. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  3. 2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
  4. 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 123
  5. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
  6. 2023PointGPT: Auto-regressively Generative Pre-training from Point Cloudscited by 115
  7. 2023Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understandingcited by 111
  8. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  9. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  10. 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
  11. 2022ALSO: Automotive Lidar Self-supervision by Occupancy estimationcited by 68
  12. 2024Point Cloud Mamba: Point Cloud Learning via State Space Modelcited by 66
  13. 2021SQN: Weakly-Supervised Semantic Segmentation of Large-Scale 3D Point Cloudscited by 64
  14. 2021Pri3D: Can 3D Priors Help 2D Representation Learning?cited by 61
  15. 2022Occupancy-MAE: Self-supervised Pre-training Large-scale LiDAR Point Clouds with Masked Occupancy Autoencoderscited by 57
  16. 2022GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Cloudscited by 57
  17. 2023PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigmcited by 56
  18. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  19. 20214DContrast: Contrastive Learning with Dynamic Correspondences for 3D Scene Understandingcited by 54
  20. 2022Unsupervised Point Cloud Representation Learning with Deep Neural Networks: A Surveycited by 54
  21. 2023PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detectioncited by 53
  22. 2024Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Drivingcited by 53
  23. 2023Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-trainingcited by 52
  24. 2022Ponder: Point Cloud Pre-training via Neural Renderingcited by 50
  25. 2022SensatUrban: Learning Semantics from Urban-Scale Photogrammetric Point Cloudscited by 49
  26. 2023Point Cloud Pre-training with Diffusion Modelscited by 47
  27. 2022Implicit Autoencoder for Point-Cloud Self-Supervised Representation Learningcited by 46
  28. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  29. 2022ProposalContrast: Unsupervised Pre-training for LiDAR-based 3D Object Detectioncited by 43
  30. 2023Three Pillars improving Vision Foundation Model Distillation for Lidarcited by 43
  31. 2021Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Modelscited by 41
  32. 2023GeoMAE: Masked Geometric Target Prediction for Self-supervised Point Cloud Pre-Trainingcited by 38
  33. 2023Self-Supervised Image-to-Point Distillation via Semantically Tolerant Contrastive Losscited by 35
  34. 2023GrowSP: Unsupervised Semantic Segmentation of 3D Point Cloudscited by 34
  35. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  36. 20244D Contrastive Superflows are Dense 3D Representation Learnerscited by 32
  37. 2023UnScene3D: Unsupervised 3D Instance Segmentation for Indoor Scenescited by 31
  38. 2022Complete-to-Partial 4D Distillation for Self-Supervised Point Cloud Sequence Representation Learningcited by 30

50 most cited of 217 All 217 (JSON)