mlsanityData to 7 Oct 2026

Self-supervised Pre-training with Masked Shape Prediction for 3D Scene Understanding

arXiv 2305.05026 ↗First posted 8 May 2023cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

Masked signal modeling has greatly advanced self-supervised pre-training for language and 2D images. However, it is still not fully explored in 3D scene understanding. Thus, this paper introduces Masked Shape Prediction (MSP), a new framework to conduct masked signal modeling in 3D scenes. MSP uses the essential 3D semantic cue, i.e., geometric shape, as the prediction target for masked points. The context-enhanced shape target consisting of explicit shape context and implicit deep shape feature is proposed to facilitate exploiting contextual cues in shape prediction. Meanwhile, the pre-training architecture in MSP is carefully designed to alleviate the masked shape leakage from point coordinates. Experiments on multiple 3D understanding tasks on both indoor and outdoor datasets demonstrate the effectiveness of MSP in learning good feature representations to consistently boost downstream performance.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 64 of its 85 references that are in the map. First posted on arXiv 8 May 2023.

Builds on

64 of 85 references.

References · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  12. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
38 more
  1. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  2. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  5. 2020Point Transformercited by 1,462
  6. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  7. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  8. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  9. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  10. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  11. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  12. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 890
  13. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  14. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  15. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  16. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  17. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  18. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  19. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  20. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  21. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  22. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  23. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  24. 2021Voxel Transformer for 3D Object Detectioncited by 247
  25. 2020PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Samplingcited by 238
  26. 2021PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detectioncited by 232
  27. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  28. 20183D-SIS: 3D Semantic Instance Segmentation of RGB-D Scanscited by 216
  29. 2021Group-Free 3D Object Detection via Transformerscited by 215
  30. 2018SO-Net: Self-Organizing Network for Point Cloud Analysiscited by 210
  31. 2020Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contextscited by 210
  32. 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
  33. 2021Spatio-temporal Self-Supervised Representation Learning for 3D Point Cloudscited by 180
  34. 2017SEGCloud: Semantic Segmentation of 3D Point Cloudscited by 178
  35. 2020OccuSeg: Occupancy-aware 3D Instance Segmentationcited by 177
  36. 2019Learning Object Bounding Boxes for 3D Instance Segmentation on Point Cloudscited by 156
  37. 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155
  38. 20203D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentationcited by 140

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 1; 2026: 3.

13 papers.

Sort:

By area:

  1. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  2. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 97
  3. 2024Point Cloud Mamba: Point Cloud Learning via State Space Modelcited by 66
  4. 2023Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generationcited by 37
  5. 2024Masked Image Modeling: A Surveycited by 28
  6. 2024NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation Learning for Neural Radiance Fieldscited by 20
  7. 2024ARKit LabelMaker: A New Scale for Indoor 3D Scene Understandingcited by 7
  8. 2024Improving Distant 3D Object Detection Using 2D Box Supervisioncited by 6
  9. 2023FATE: Feature-Agnostic Transformer-based Encoder for learning generalized embedding spaces in flow cytometry datacited by 2
  10. 2026PointCSP: Cross-Sample Semantic Propagation and Stability Preservation in Self-Supervised Point Cloud Learningcited by 1
  11. 2025CoL3D: Collaborative Learning of Single-view Depth and Camera Intrinsics for Metric 3D Shape Recoverycited by 0
  12. 2026SR-JEPA: Learning Predictive Latent State in 3D Scenescited by 0
1 more
  1. 2026TokenMatch: 3D Mesh Correspondence Transformer with Curvature-Guided Tokenisationcited by 0