ExpPoint-MAE: Better interpretability and performance for self-supervised point cloud transformers
Not on the map. Why?
Abstract (arXiv)
In this paper we delve into the properties of transformers, attained through self-supervision, in the point cloud domain. Specifically, we evaluate the effectiveness of Masked Autoencoding as a pretraining scheme, and explore Momentum Contrast as an alternative. In our study we investigate the impact of data quantity on the learned features, and uncover similarities in the transformer's behavior across domains. Through comprehensive visualiations, we observe that the transformer learns to attend to semantically meaningful regions, indicating that pretraining leads to a better understanding of the underlying geometry. Moreover, we examine the finetuning process and its effect on the learned representations. Based on that, we devise an unfreezing strategy which consistently outperforms our baseline without introducing any other modifications to the model or the training pipeline, and achieve state-of-the-art results in the classification task among transformer models.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 52 of its 71 references that are in the map. First posted on arXiv 19 Jun 2023.
Builds on
52 of 71 references.
References · 52
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
38 more
- 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
- 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
- 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
- 2019Similarity of Neural Network Representations Revisitedcited by 1,755
- 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
- 2020Point Transformercited by 1,470
- 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,205
- 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
- 2020PCT: Point cloud transformercited by 732
- 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
- 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
- 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
- 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
- 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
- 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 565
- 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
- 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 504
- 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
- 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 379
- 2019Relation-Shape Convolutional Neural Network for Point Cloud Analysiscited by 346
- 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 264
- 2018SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filterscited by 261
- 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
- 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 245
- 2016Volumetric and Multi-View CNNs for Object Classification on 3D Datacited by 240
- 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
- 2021Walk in the Cloud: Learning Curves for Point Clouds Shape Analysiscited by 172
- 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155
- 2022ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyondcited by 117
- 2021Pre-training without Natural Imagescited by 100
- 2021Guided Point Contrastive Learning for Semi-supervised Point Cloud Semantic Segmentationcited by 87
- 2021Self-Contrastive Learning with Hard Negative Sampling for Self-supervised Point Cloud Learningcited by 52
- 2021PVT: Point-Voxel Transformer for Point Cloud Learningcited by 43
- 2021Zero-Shot Learning on 3D Point Cloud Objects and Beyondcited by 38
- 2018PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognitioncited by 36
- 2021PvDeConv: Point-Voxel Deconvolution for Autoencoding CAD Construction in 3Dcited by 20
50 most cited of 52 All 52 (JSON)
Built on by
Citing papers per year: 2024: 3.
3 papers.