mlsanityData to 7 Oct 2026

Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling

arXiv 2111.14819 ↗First posted 29 Nov 2021cs.CV cs.AI cs.LGcited by 647

Not on the map. Why?

Cited by 45 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Point-BERT, a new paradigm for learning Transformers to generalize the concept of BERT to 3D point cloud. Inspired by BERT, we devise a Masked Point Modeling (MPM) task to pre-train point cloud Transformers. Specifically, we first divide a point cloud into several local point patches, and a point cloud Tokenizer with a discrete Variational AutoEncoder (dVAE) is designed to generate discrete point tokens containing meaningful local information. Then, we randomly mask out some patches of input point clouds and feed them into the backbone Transformers. The pre-training objective is to recover the original point tokens at the masked locations under the supervision of point tokens obtained by the Tokenizer. Extensive experiments demonstrate that the proposed BERT-style pre-training strategy significantly improves the performance of standard point cloud Transformers. Equipped with our pre-training strategy, we show that a pure Transformer architecture attains 93.8% accuracy on ModelNet40 and 83.1% accuracy on the hardest setting of ScanObjectNN, surpassing carefully designed point cloud models with much fewer hand-made designs. We also demonstrate that the representations learned by Point-BERT transfer well to new tasks and domains, where our models largely advance the state-of-the-art of few-shot point cloud classification task. The code and pre-trained models are available at https://github.com/lulutang0608/Point-BERT

Key facts. Cited by 647 papers in the map (data through 7 Oct 2026). Builds on 59 of its 74 references that are in the map. First posted on arXiv 29 Nov 2021.

Builds on

59 of 74 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  12. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  10. 2017Decoupled Weight Decay Regularizationcited by 1,722
  11. 2020Point Transformercited by 1,462
  12. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  13. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  14. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  15. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  16. 2021Transformers in Vision: A Surveycited by 1,009
  17. 2016Deep Networks with Stochastic Depthcited by 973
  18. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  19. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  20. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  21. 2018Image Transformercited by 770
  22. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  23. 2020PCT: Point cloud transformercited by 730
  24. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  25. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  26. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 417
  27. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  28. 2021PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformerscited by 284
  29. 2018SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filterscited by 261
  30. 2016Learning Representations for Automatic Colorizationcited by 251
  31. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  32. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  33. 2018SO-Net: Self-Organizing Network for Point Cloud Analysiscited by 210
  34. 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
  35. 2020Learning Joint Spatial-Temporal Transformations for Video Inpaintingcited by 162
  36. 2016Discrete Variational Autoencoderscited by 124
  37. 2021Self-Supervised Learning with Swin Transformerscited by 121
  38. 2019DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processingcited by 104

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 56; 2023: 145; 2024: 158; 2025: 159; 2026: 129.

647 papers.

Sort:

By area:

  1. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  2. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  3. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  4. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  5. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  6. 2024Transolver: A Fast Transformer Solver for PDEs on General Geometriescited by 299
  7. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  8. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  9. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  10. 2023MeshGPT: Generating Triangle Meshes with Decoder-Only Transformerscited by 254
  11. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 231
  12. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 219
38 more
  1. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  2. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  3. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  4. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  5. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  6. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
  7. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  8. 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155
  9. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  10. 2023Multimodal Industrial Anomaly Detection via Hybrid Fusioncited by 138
  11. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  12. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  13. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  14. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  15. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
  16. 2023PointGPT: Auto-regressively Generative Pre-training from Point Cloudscited by 115
  17. 2023FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimationcited by 114
  18. 2023Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understandingcited by 111
  19. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  20. 2025Sonata: Self-Supervised Learning of Reliable Point Representationscited by 108
  21. 2022ShapeFormer: Transformer-based Shape Completion via Sparse Representationcited by 106
  22. 2023OctFormer: Octree-based Transformers for 3D Point Cloudscited by 105
  23. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  24. 20223DILG: Irregular Latent Grids for 3D Generative Modelingcited by 92
  25. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  26. 2023Real3D-AD: A Dataset of Point Cloud Anomaly Detectioncited by 87
  27. 2025Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assetscited by 85
  28. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  29. 2023RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Drivingcited by 76
  30. 2024MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learningcited by 76
  31. 2022P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Promptingcited by 74
  32. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 74
  33. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  34. 2023Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoderscited by 72
  35. 2022Benchmarking and Analyzing Point Cloud Classification under Corruptionscited by 70
  36. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69
  37. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 69
  38. 2023Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Modelscited by 67

50 most cited of 647 All 647 (JSON)