mlsanityData to 7 Oct 2026

Take-A-Photo: 3D-to-2D Generative Pre-training of Point Cloud Models

arXiv 2307.14971 ↗First posted 27 Jul 2023cs.CV cs.AI cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

With the overwhelming trend of mask image modeling led by MAE, generative pre-training has shown a remarkable potential to boost the performance of fundamental models in 2D vision. However, in 3D vision, the over-reliance on Transformer-based backbones and the unordered nature of point clouds have restricted the further development of generative pre-training. In this paper, we propose a novel 3D-to-2D generative pre-training method that is adaptable to any point cloud model. We propose to generate view images from different instructed poses via the cross-attention mechanism as the pre-training scheme. Generating view images has more precise supervision than its point cloud counterpart, thus assisting 3D backbones to have a finer comprehension of the geometrical structure and stereoscopic relations of the point cloud. Experimental results have proved the superiority of our proposed 3D-to-2D generative pre-training over previous pre-training methods. Our method is also effective in boosting the performance of architecture-oriented approaches, achieving state-of-the-art performance when fine-tuning on ScanObjectNN classification and ShapeNetPart segmentation tasks. Code is available at https://github.com/wangzy22/TAP.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 54 of its 65 references that are in the map. First posted on arXiv 27 Jul 2023.

Builds on

54 of 65 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  12. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
38 more
  1. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  3. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  6. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  7. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  8. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  9. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  10. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  11. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  12. 2021Scaling Vision Transformerscited by 993
  13. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  14. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  15. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  16. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 564
  17. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  18. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  19. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  20. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  21. 2022Point Transformer V2: Grouped Vector Attention and Partition-based Poolingcited by 384
  22. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 376
  23. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 340
  24. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  25. 2021PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformerscited by 284
  26. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  27. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  28. 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
  29. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
  30. 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155
  31. 2021Revisiting Point Cloud Shape Classification with a Simple and Effective Baselinecited by 145
  32. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  33. 2017Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Modelscited by 142
  34. 2020MVTN: Multi-View Transformation Network for 3D Shape Recognitioncited by 140
  35. 2019Morphing and Sampling Network for Dense Point Cloud Completioncited by 132
  36. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
  37. 2020Self-Supervised Learning for Domain Adaptation on Point-Cloudscited by 99
  38. 2022Surface Representation for Point Cloudscited by 84

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 9; 2026: 5.

25 papers.

Sort:

By area:

  1. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  2. 2023Point Cloud Pre-training with Diffusion Modelscited by 47
  3. 2024PCP-MAE: Learning to Predict Centers for Point Masked Autoencoderscited by 23
  4. 2024Bringing Masked Autoencoders Explicit Contrastive Properties for Point Cloud Self-Supervised Learningcited by 16
  5. 2024PointNet with KAN versus PointNet with MLP for 3D Classification and Segmentation of Point Setscited by 15
  6. 20253D CoCa: Contrastive Learners are 3D Captionerscited by 13
  7. 2024Shape2Scene: 3D Scene Representation Learning Through Pre-training on Shape Datacited by 8
  8. 2025Towards More Diverse and Challenging Pre-training for Point Cloud Learning: Self-Supervised Cross Reconstruction with Decoupled Viewscited by 7
  9. 2025UniPre3D: Unified Pre-training of 3D Point Cloud Models with Cross-Modal Gaussian Splattingcited by 6
  10. 2025Harnessing Text-to-Image Diffusion Models for Point Cloud Self-Supervised Learningcited by 3
  11. 20263D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligencecited by 3
  12. 2023Point Cloud Self-supervised Learning via 3D to Multi-view Masked Learnercited by 2
13 more
  1. 2024Pre-training Point Cloud Compact Model with Partial-aware Reconstructioncited by 2
  2. 2025Masked Clustering Prediction for Unsupervised Point Cloud Pre-trainingcited by 2
  3. 2024Triple Point Maskingcited by 1
  4. 2024Pic@Point: Cross-Modal Learning by Local and Global Point-Picture Correspondencecited by 1
  5. 2025DAP-MAE: Domain-Adaptive Point Cloud Masked Autoencoder for Effective Cross-Domain Learningcited by 1
  6. 2026PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Modelscited by 1
  7. 2026Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Frameworkcited by 1
  8. 2026From Extrinsic to Intrinsic: Geodesic-Guided Representation Learning for 3D Geometric Datacited by 1
  9. 2024PointCG: Self-supervised Point Cloud Learning via Joint Completion and Generationcited by 0
  10. 2025Towards Training-Free Open-World Classification with 3D Generative Modelscited by 0
  11. 2025PointVDP: Learning View-Dependent Projection by Fireworks Rays for 3D Point Cloud Segmentationcited by 0
  12. 2025Dual-Branch Center-Surrounding Contrast: Rethinking Contrastive Learning for 3D Point Cloudscited by 0
  13. 2026GAPrompt++: Multi-Granular Geometry-Aware Point Cloud Prompt for 3D Vision Modelcited by 0