# Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models

arXiv: https://arxiv.org/abs/2106.04180 · HTML: https://mlsanity.com/paper/2106.04180/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 98 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 30 of 67)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)

## Built on by (most cited first, 30 of 41)

- [CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding](https://mlsanity.com/paper/2203.00680/) (2022)
- [Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following](https://mlsanity.com/paper/2309.00615/) (2023)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP](https://mlsanity.com/paper/2303.04748/) (2023)
- [SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detection](https://mlsanity.com/paper/2304.14340/) (2023)
- [OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation](https://mlsanity.com/paper/2309.00616/) (2023)
- [Open-Vocabulary Point-Cloud Object Detection without 3D Annotation](https://mlsanity.com/paper/2304.00788/) (2023)
- [X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning](https://mlsanity.com/paper/2203.00843/) (2022)
- [ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidance](https://mlsanity.com/paper/2303.16894/) (2023)
- [GrowSP: Unsupervised Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/2305.16404/) (2023)
- [Take-A-Photo: 3D-to-2D Generative Pre-training of Point Cloud Models](https://mlsanity.com/paper/2307.14971/) (2023)
- [ODIN: A Single Model for 2D and 3D Segmentation](https://mlsanity.com/paper/2401.02416/) (2024)
- [Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detection](https://mlsanity.com/paper/2309.09456/) (2023)
- [Any2Point: Empowering Any-modality Large Models for Efficient 3D Understanding](https://mlsanity.com/paper/2404.07989/) (2024)
- [Hierarchical Point-based Active Learning for Semi-supervised Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2308.11166/) (2023)
- [FM-OV3D: Foundation Model-based Cross-modal Knowledge Blending for Open-Vocabulary 3D Detection](https://mlsanity.com/paper/2312.14465/) (2023)
- [EPCL: Frozen CLIP Transformer is An Efficient Point Cloud Encoder](https://mlsanity.com/paper/2212.04098/) (2022)
- [ImLoveNet: Misaligned Image-supported Registration Network for Low-overlap Point Cloud Pairs](https://mlsanity.com/paper/2207.00826/) (2022)
- [PointDC:Unsupervised Semantic Segmentation of 3D Point Clouds via Cross-modal Distillation and Super-Voxel Clustering](https://mlsanity.com/paper/2304.08965/) (2023)
- [3DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4V](https://mlsanity.com/paper/2312.09738/) (2023)
- [What Matters to You? Towards Visual Representation Alignment for Robot Learning](https://mlsanity.com/paper/2310.07932/) (2023)
- [PCR-CG: Point Cloud Registration via Deep Explicit Color and Geometry](https://mlsanity.com/paper/2302.14418/) (2023)
- [Invariant Training 2D-3D Joint Hard Samples for Few-Shot Point Cloud Recognition](https://mlsanity.com/paper/2308.09694/) (2023)
- [On the Implicit Bias Towards Minimal Depth of Deep Neural Networks](https://mlsanity.com/paper/2202.09028/) (2022)
- [CP3: Channel Pruning Plug-in for Point-based Networks](https://mlsanity.com/paper/2303.13097/) (2023)
- [Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities](https://mlsanity.com/paper/2401.14405/) (2024)
- [Regress Before Construct: Regress Autoencoder for Point Cloud Self-supervised Learning](https://mlsanity.com/paper/2310.03670/) (2023)
- [CrossVideo: Self-supervised Cross-modal Contrastive Learning for Point Cloud Video Understanding](https://mlsanity.com/paper/2401.09057/) (2024)
- [CrossJEPA: Cross-Modal Joint-Embedding Predictive Architecture for Efficient 3D Representation Learning from 2D Images](https://mlsanity.com/paper/2511.18424/) (2025)
- [Multi-View Representation is What You Need for Point-Cloud Pre-Training](https://mlsanity.com/paper/2306.02558/) (2023)

## Abstract (arXiv)

3D point-clouds and 2D images are different visual representations of the physical world. While human vision can understand both representations, computer vision models designed for 2D image and 3D point-cloud understanding are quite different. Our paper explores the potential of transferring 2D model architectures and weights to understand 3D point-clouds, by empirically investigating the feasibility of the transfer, the benefits of the transfer, and shedding light on why the transfer works. We discover that we can indeed use the same architecture and pretrained weights of a neural net model to understand both images and point-clouds. Specifically, we transfer the image-pretrained model to a point-cloud model by copying or inflating the weights. We find that finetuning the transformed image-pretrained models (FIP) with minimal efforts -- only on input, output, and normalization layers -- can achieve competitive performance on 3D point-cloud classification, beating a wide range of point-cloud models that adopt task-specific architectures and use a variety of tricks. When finetuning the whole model, the performance improves even further. Meanwhile, FIP improves data efficiency, reaching up to 10.0 top-1 accuracy percent on few-shot classification. It also speeds up the training of point-cloud models by up to 11.1x for a target accuracy (e.g., 90 % accuracy). Lastly, we provide an explanation of the image to point-cloud transfer from the aspect of neural collapse. The code is available at: urlhttps://github.com/chenfengxu714/image2point.