mlsanityData to 7 Oct 2026

PointGPT: Auto-regressively Generative Pre-training from Point Clouds

arXiv 2305.11487 ↗First posted 19 May 2023cs.CVcited by 115

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) based on the generative pre-training transformer (GPT) have demonstrated remarkable effectiveness across a diverse range of downstream tasks. Inspired by the advancements of the GPT, we present PointGPT, a novel approach that extends the concept of GPT to point clouds, addressing the challenges associated with disorder properties, low information density, and task gaps. Specifically, a point cloud auto-regressive generation task is proposed to pre-train transformer models. Our method partitions the input point cloud into multiple point patches and arranges them in an ordered sequence based on their spatial proximity. Then, an extractor-generator based transformer decoder, with a dual masking strategy, learns latent representations conditioned on the preceding point patches, aiming to predict the next one in an auto-regressive manner. Our scalable approach allows for learning high-capacity models that generalize well, achieving state-of-the-art performance on various downstream tasks. In particular, our approach achieves classification accuracies of 94.9% on the ModelNet40 dataset and 93.4% on the ScanObjectNN dataset, outperforming all other transformer models. Furthermore, our method also attains new state-of-the-art accuracies on all four few-shot learning benchmarks.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Builds on 46 of its 70 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

46 of 70 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2022Generative Adversarial Networkscited by 5,603
  8. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  11. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
34 more
  1. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  2. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  3. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  4. 2021Scaling Vision Transformerscited by 993
  5. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  6. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  7. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  8. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  9. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  10. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  11. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  12. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  13. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  14. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  15. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  16. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 376
  17. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  18. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  19. 2019Relation-Shape Convolutional Neural Network for Point Cloud Analysiscited by 346
  20. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  21. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  22. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  23. 2018SO-Net: Self-Organizing Network for Point Cloud Analysiscited by 210
  24. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  25. 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
  26. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
  27. 2017Semantic3D.net: A new Large-scale Point Cloud Classification Benchmarkcited by 158
  28. 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155
  29. 2020MVTN: Multi-View Transformation Network for 3D Shape Recognitioncited by 140
  30. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
  31. 2018Multiresolution Tree Networks for 3D Point Cloud Processingcited by 91
  32. 2022P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Promptingcited by 74
  33. 2020Self-supervised Modal and View Invariant Feature Learningcited by 20
  34. 2020From Image Collections to Point Clouds with Self-supervised Shape and Pose Networkscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 33; 2025: 46; 2026: 30.

115 papers.

Sort:

By area:

  1. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  2. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  3. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
  4. 2024Point Cloud Mamba: Point Cloud Learning via State Space Modelcited by 66
  5. 2024Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detectioncited by 66
  6. 2024Mamba3D: Enhancing Local Features for 3D Point Cloud Analysis via State Space Modelcited by 57
  7. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  8. 2024Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learningcited by 43
  9. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 34
  10. 2024Point-JEPA: A Joint Embedding Predictive Architecture for Self-Supervised Learning on Point Cloudcited by 29
  11. 2024PCP-MAE: Learning to Predict Centers for Point Masked Autoencoderscited by 23
  12. 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latentscited by 19
38 more
  1. 2026Align then Adapt: Rethinking Parameter-Efficient Transfer Learning in 4D Perceptioncited by 17
  2. 2024Bringing Masked Autoencoders Explicit Contrastive Properties for Point Cloud Self-Supervised Learningcited by 16
  3. 2025Exploring the Potential of Encoder-free Architectures in 3D LMMscited by 16
  4. 2024RI-MAE: Rotation-Invariant Masked AutoEncoders for Self-Supervised Point Cloud Representation Learningcited by 15
  5. 2025TripNet: Learning Large-scale High-fidelity 3D Car Aerodynamics with Triplane Networkscited by 14
  6. 2025PointLoRA: Low-Rank Adaptation with Token Selection for Point Cloud Learningcited by 13
  7. 2025GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Modelcited by 13
  8. 2024LCM: Locally Constrained Compact Point Cloud Model for Masked Point Modelingcited by 12
  9. 2024GPSFormer: A Global Perception and Local Structure Fitting-based Transformer for Point Cloud Understandingcited by 12
  10. 2025Spectral Informed Mamba for Robust Point Cloud Processingcited by 10
  11. 2025Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Reviewcited by 10
  12. 2025PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learningcited by 10
  13. 2026PointRFT: Explicit Reinforcement Fine-tuning for Point Cloud Few-shot Learningcited by 10
  14. 2023General Point Model with Autoencoding and Autoregressivecited by 9
  15. 2024ConDense: Consistent 2D/3D Pre-training for Dense and Sparse Features from Multi-View Imagescited by 9
  16. 2025PMA: Towards Parameter-Efficient Point Cloud Understanding via Point Mamba Adaptercited by 9
  17. 2025PointNSP: Autoregressive 3D Point Cloud Generation with Next-Scale Level-of-Detail Predictioncited by 9
  18. 2024Shape2Scene: 3D Scene Representation Learning Through Pre-training on Shape Datacited by 8
  19. 2024Efficient and Scalable Point Cloud Generation with Sparse Point-Voxel Diffusion Modelscited by 7
  20. 2025Towards More Diverse and Challenging Pre-training for Point Cloud Learning: Self-Supervised Cross Reconstruction with Decoupled Viewscited by 7
  21. 2023AllSpark: A Multimodal Spatio-Temporal General Intelligence Model with Ten Modalities via Language as a Reference Frameworkcited by 6
  22. 2025MoST: Efficient Monarch Sparse Tuning for 3D Representation Learningcited by 6
  23. 2025MODEM: A Morton-Order Degradation Estimation Mechanism for Adverse Weather Image Recoverycited by 6
  24. 2025UniPre3D: Unified Pre-training of 3D Point Cloud Models with Cross-Modal Gaussian Splattingcited by 6
  25. 2026Mantis: Mamba-native Tuning is Efficient for 3D Point Cloud Foundation Modelscited by 6
  26. 2024Multi-modal Relation Distillation for Unified 3D Representation Learningcited by 5
  27. 2024Point Cloud Mixture-of-Domain-Experts Model for 3D Self-supervised Learningcited by 5
  28. 2024PointPatchRL -- Masked Reconstruction Improves Reinforcement Learning on Point Cloudscited by 5
  29. 2025CASL: Curvature-Augmented Self-supervised Learning for 3D Anomaly Detectioncited by 5
  30. 2026Tri-Efficient Transfer Learning for Point Cloud Videoscited by 5
  31. 2024MaskLRF: Self-supervised Pretraining via Masked Autoencoding of Local Reference Frames for Rotation-invariant 3D Point Set Analysiscited by 4
  32. 2025Point-Cache: Test-time Dynamic and Hierarchical Cache for Robust and Generalizable Point Cloud Analysiscited by 4
  33. 2025PointMAC: Meta-Learned Adaptation for Robust Test-Time Point Cloud Completioncited by 4
  34. 2025CrossJEPA: Cross-Modal Joint-Embedding Predictive Architecture for Efficient 3D Representation Learning from 2D Imagescited by 4
  35. 2024A Unified Framework for Human-centric Point Cloud Video Understandingcited by 3
  36. 2024GeoMask3D: Geometrically Informed Mask Selection for Self-Supervised Point Cloud Learning in 3Dcited by 3
  37. 2024NIMBA: Towards Robust and Principled Processing of Point Clouds With SSMscited by 3
  38. 2025StruMamba3D: Exploring Structural Mamba for Self-supervised Point Cloud Representation Learningcited by 3

50 most cited of 115 All 115 (JSON)