Bridging the Dimensionality Gap: A Taxonomy and Survey of 2D Vision Model Adaptation for 3D Analysis
Not on the map. Why?
Abstract (arXiv)
The remarkable success of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) in 2D vision has spurred significant research in extending these architectures to the complex domain of 3D analysis. Yet, a core challenge arises from a fundamental dichotomy between the regular, dense grids of 2D images and the irregular, sparse nature of 3D data such as point clouds and meshes. This survey provides a comprehensive review and a unified taxonomy of adaptation strategies that bridge this gap, classifying them into three families: (1) Data-centric methods that project 3D data into 2D formats to leverage off-the-shelf 2D models, (2) Architecture-centric methods that design intrinsic 3D networks, and (3) Hybrid methods, which synergistically combine the two modeling paradigms to benefit from both rich visual priors of large 2D datasets and explicit geometric reasoning of 3D models. Through this framework, we qualitatively analyze the fundamental trade-offs between these families concerning computational complexity, reliance on large-scale pre-training, and the preservation of geometric inductive biases. We discuss key open challenges and outline promising future research directions, including the development of 3D foundation models, advancements in self-supervised learning (SSL) for geometric data, and the deeper integration of multi-modal signals.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 61 of its 90 references that are in the map. First posted on arXiv 3 Apr 2026.
Builds on
61 of 90 references.
References · 61
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2017Graph Attention Networkscited by 8,854
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
- 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
- 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
- 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
- 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
- 2020Point Transformercited by 1,462
- 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
38 more
- 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
- 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
- 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
- 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
- 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
- 2018PointCNN: Convolution On X-Transformed Pointscited by 764
- 2020PCT: Point cloud transformercited by 730
- 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
- 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 564
- 2019Deep Learning for 3D Point Clouds: A Surveycited by 557
- 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
- 2018Spherical CNNscited by 452
- 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
- 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
- 2020Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloudcited by 335
- 2019Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challengescited by 313
- 2020PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentationcited by 274
- 2025SAM 3D: 3Dfy Anything in Imagescited by 230
- 2022DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detectioncited by 224
- 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
- 2017SEGCloud: Semantic Segmentation of 3D Point Cloudscited by 178
- 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
- 20203D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detectioncited by 176
- 2017PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimationcited by 170
- 2021Multimodal Virtual Point 3D Detectioncited by 166
- 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 166
- 2020EPNet: Enhancing Point Features with Image Semantics for 3D Object Detectioncited by 164
- 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 157
- 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
- 2022Solving 3D Inverse Problems using Pre-trained 2D Diffusion Modelscited by 130
- 2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
- 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
- 2017Deep Projective 3D Semantic Segmentationcited by 81
- 2022P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Promptingcited by 74
- 2022AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detectioncited by 70
- 2021SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representationscited by 55
- 2023PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detectioncited by 53
- 2023Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-trainingcited by 52
50 most cited of 61 All 61 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026A Cookbook of 3D Vision: Data, Learning Paradigms, and Application
- 2024ImageNet3D: Towards General-Purpose Object-Level 3D Understanding
- 2022Deep Generative Models on 3D Representations: A Survey
- 2026Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective
- 2024Diffusion Models in 3D Vision: A Survey
- 20243D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation
- 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors
- 2026Video Generative Models as Geometry Learner