mlsanityData to 14 Oct 2026

Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models

arXiv 2106.04180 ↗First posted 8 Jun 2021cs.CV cs.AI cs.ROcited by 41

Not on the map. Why?

Abstract (arXiv)

3D point-clouds and 2D images are different visual representations of the physical world. While human vision can understand both representations, computer vision models designed for 2D image and 3D point-cloud understanding are quite different. Our paper explores the potential of transferring 2D model architectures and weights to understand 3D point-clouds, by empirically investigating the feasibility of the transfer, the benefits of the transfer, and shedding light on why the transfer works. We discover that we can indeed use the same architecture and pretrained weights of a neural net model to understand both images and point-clouds. Specifically, we transfer the image-pretrained model to a point-cloud model by copying or inflating the weights. We find that finetuning the transformed image-pretrained models (FIP) with minimal efforts -- only on input, output, and normalization layers -- can achieve competitive performance on 3D point-cloud classification, beating a wide range of point-cloud models that adopt task-specific architectures and use a variety of tricks. When finetuning the whole model, the performance improves even further. Meanwhile, FIP improves data efficiency, reaching up to 10.0 top-1 accuracy percent on few-shot classification. It also speeds up the training of point-cloud models by up to 11.1x for a target accuracy (e.g., 90 % accuracy). Lastly, we provide an explanation of the image to point-cloud transfer from the aspect of neural collapse. The code is available at: urlhttps://github.com/chenfengxu714/image2point.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 67 of its 98 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

67 of 98 references.

References · 67

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  9. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
38 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  2. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
  3. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  4. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  5. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  6. 2020Point Transformercited by 1,470
  7. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  8. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  9. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  10. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  11. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  12. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  13. 2016Generative Adversarial Text to Image Synthesiscited by 735
  14. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  15. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  16. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  17. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 657
  18. 2021ImageNet-21K Pretraining for the Massescited by 613
  19. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 565
  20. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 504
  21. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 457
  22. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 423
  23. 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 406
  24. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  25. 2017SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloudcited by 232
  26. 2018SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaptation for Road-Object Segmentation from a LiDAR Point Cloudcited by 228
  27. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  28. 2018SO-Net: Self-Organizing Network for Point Cloud Analysiscited by 210
  29. 2020Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contextscited by 210
  30. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 198
  31. 2020SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentationcited by 170
  32. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  33. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  34. 2017Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Modelscited by 142
  35. 2017Pointwise Convolutional Neural Networkscited by 127
  36. 20183DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentationcited by 118
  37. 2020Cylinder3D: An Effective 3D Framework for Driving-scene LiDAR Semantic Segmentationcited by 112
  38. 2019DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processingcited by 104

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 21; 2024: 9; 2025: 4; 2026: 2.

41 papers.

Sort:

By area:

  1. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  2. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  3. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  4. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  5. 2023SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detectioncited by 83
  6. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  7. 2023Open-Vocabulary Point-Cloud Object Detection without 3D Annotationcited by 81
  8. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  9. 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
  10. 2023GrowSP: Unsupervised Semantic Segmentation of 3D Point Cloudscited by 34
  11. 2023Take-A-Photo: 3D-to-2D Generative Pre-training of Point Cloud Modelscited by 25
  12. 2024ODIN: A Single Model for 2D and 3D Segmentationcited by 25
29 more
  1. 2023Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detectioncited by 24
  2. 2024Any2Point: Empowering Any-modality Large Models for Efficient 3D Understandingcited by 23
  3. 2023Hierarchical Point-based Active Learning for Semi-supervised Point Cloud Semantic Segmentationcited by 21
  4. 2023FM-OV3D: Foundation Model-based Cross-modal Knowledge Blending for Open-Vocabulary 3D Detectioncited by 19
  5. 2022EPCL: Frozen CLIP Transformer is An Efficient Point Cloud Encodercited by 18
  6. 2022ImLoveNet: Misaligned Image-supported Registration Network for Low-overlap Point Cloud Pairscited by 17
  7. 2023PointDC:Unsupervised Semantic Segmentation of 3D Point Clouds via Cross-modal Distillation and Super-Voxel Clusteringcited by 15
  8. 20233DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4Vcited by 15
  9. 2023What Matters to You? Towards Visual Representation Alignment for Robot Learningcited by 14
  10. 2023PCR-CG: Point Cloud Registration via Deep Explicit Color and Geometrycited by 13
  11. 2023Invariant Training 2D-3D Joint Hard Samples for Few-Shot Point Cloud Recognitioncited by 13
  12. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  13. 2023CP3: Channel Pruning Plug-in for Point-based Networkscited by 10
  14. 2024Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalitiescited by 7
  15. 2023Regress Before Construct: Regress Autoencoder for Point Cloud Self-supervised Learningcited by 6
  16. 2024CrossVideo: Self-supervised Cross-modal Contrastive Learning for Point Cloud Video Understandingcited by 6
  17. 2025CrossJEPA: Cross-Modal Joint-Embedding Predictive Architecture for Efficient 3D Representation Learning from 2D Imagescited by 4
  18. 2023Multi-View Representation is What You Need for Point-Cloud Pre-Trainingcited by 3
  19. 2023Point Clouds Are Specialized Images: A Knowledge Transfer Approach for 3D Understandingcited by 3
  20. 2024Scaling Manipulation Learning with Visual Kinematic Chain Predictioncited by 2
  21. 2024Pic@Point: Cross-Modal Learning by Local and Global Point-Picture Correspondencecited by 1
  22. 20252D-3D Attention and Entropy for Pose Robust 2D Facial Recognitioncited by 1
  23. 2026AdaptSplat: Adapting Vision Foundation Models for Feed-Forward 3D Gaussian Splattingcited by 1
  24. 2024Let's Go Shopping (LGS) -- Web-Scale Image-Text Dataset for Visual Concept Understandingcited by 0
  25. 2024ELiTe: Efficient Image-to-LiDAR Knowledge Transfer for Semantic Segmentationcited by 0
  26. 2024Boosting Cross-Domain Point Classification via Distilling Relational Priors from 2D Transformerscited by 0
  27. 2025Multi-modal Multi-task Pre-training for Improved Point Cloud Understandingcited by 0
  28. 2025Domain Adaptation-Based Crossmodal Knowledge Distillation for 3D Semantic Segmentationcited by 0
  29. 2026Bridging the Dimensionality Gap: A Taxonomy and Survey of 2D Vision Model Adaptation for 3D Analysiscited by 0