mlsanityData to 7 Oct 2026

Bridging the Dimensionality Gap: A Taxonomy and Survey of 2D Vision Model Adaptation for 3D Analysis

arXiv 2604.03334 ↗First posted 3 Apr 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

The remarkable success of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) in 2D vision has spurred significant research in extending these architectures to the complex domain of 3D analysis. Yet, a core challenge arises from a fundamental dichotomy between the regular, dense grids of 2D images and the irregular, sparse nature of 3D data such as point clouds and meshes. This survey provides a comprehensive review and a unified taxonomy of adaptation strategies that bridge this gap, classifying them into three families: (1) Data-centric methods that project 3D data into 2D formats to leverage off-the-shelf 2D models, (2) Architecture-centric methods that design intrinsic 3D networks, and (3) Hybrid methods, which synergistically combine the two modeling paradigms to benefit from both rich visual priors of large 2D datasets and explicit geometric reasoning of 3D models. Through this framework, we qualitatively analyze the fundamental trade-offs between these families concerning computational complexity, reliance on large-scale pre-training, and the preservation of geometric inductive biases. We discuss key open challenges and outline promising future research directions, including the development of 3D foundation models, advancements in self-supervised learning (SSL) for geometric data, and the deeper integration of multi-modal signals.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 61 of its 90 references that are in the map. First posted on arXiv 3 Apr 2026.

Builds on

61 of 90 references.

References · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2017Graph Attention Networkscited by 8,854
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  8. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  9. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  10. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
  11. 2020Point Transformercited by 1,462
  12. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
38 more
  1. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  2. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  3. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  4. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  5. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  6. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  7. 2020PCT: Point cloud transformercited by 730
  8. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  9. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 564
  10. 2019Deep Learning for 3D Point Clouds: A Surveycited by 557
  11. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  12. 2018Spherical CNNscited by 452
  13. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  14. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  15. 2020Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloudcited by 335
  16. 2019Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challengescited by 313
  17. 2020PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentationcited by 274
  18. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  19. 2022DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detectioncited by 224
  20. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  21. 2017SEGCloud: Semantic Segmentation of 3D Point Cloudscited by 178
  22. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  23. 20203D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detectioncited by 176
  24. 2017PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimationcited by 170
  25. 2021Multimodal Virtual Point 3D Detectioncited by 166
  26. 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 166
  27. 2020EPNet: Enhancing Point Features with Image Semantics for 3D Object Detectioncited by 164
  28. 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 157
  29. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  30. 2022Solving 3D Inverse Problems using Pre-trained 2D Diffusion Modelscited by 130
  31. 2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
  32. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
  33. 2017Deep Projective 3D Semantic Segmentationcited by 81
  34. 2022P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Promptingcited by 74
  35. 2022AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detectioncited by 70
  36. 2021SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representationscited by 55
  37. 2023PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detectioncited by 53
  38. 2023Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-trainingcited by 52

50 most cited of 61 All 61 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Cookbook of 3D Vision: Data, Learning Paradigms, and Application
  2. 2024ImageNet3D: Towards General-Purpose Object-Level 3D Understanding
  3. 2022Deep Generative Models on 3D Representations: A Survey
  4. 2026Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective
  5. 2024Diffusion Models in 3D Vision: A Survey
  6. 20243D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation
  7. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors
  8. 2026Video Generative Models as Geometry Learner