A Cookbook of 3D Vision: Data, Learning Paradigms, and Application
Not on the map. Why?
Abstract (arXiv)
3D vision has rapidly evolved, driven by increasingly diverse data representations, learning paradigms, and modeling strategies. Yet the field remains fragmented across representations and benchmarks, making it difficult to develop unified perspectives on efficiency, fidelity, and scalability. This work provides a data-centric taxonomy of 3D vision that connects geometric representations, datasets, learning frameworks, and applications within a single conceptual map. We begin by analysing the principal structural representations of 3D data--point clouds, meshes, voxels, and 3D Gaussians--along with their acquisition pipelines. We then examine how dataset design, benchmark construction, and supervision regimes shape recent advances, spanning 2D-supervised 3D learning, implicit neural representations, and 4D world modeling. Through this integrative lens, we clarify the relationships among representations, learning paradigms, and downstream tasks in reconstruction, generation, and video modeling, offering a consolidated view of emerging trends toward balancing efficiency and fidelity and toward multimodal geometric grounding.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 96 of its 156 references that are in the map. First posted on arXiv 2 Jun 2026.
Builds on
96 of 156 references.
References · 96
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
- 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
- 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
- 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
- 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
- 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
- 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
- 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
- 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
- 2020Point Transformercited by 1,462
- 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
38 more
- 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
- 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
- 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
- 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,144
- 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
- 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
- 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
- 2020Neural Sparse Voxel Fieldscited by 959
- 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
- 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
- 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
- 2016ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Camerascited by 801
- 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
- 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
- 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
- 2020PCT: Point cloud transformercited by 730
- 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
- 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
- 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
- 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 626
- 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
- 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
- 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
- 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
- 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 514
- 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
- 2019BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networkscited by 487
- 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
- 2017Neural 3D Mesh Renderercited by 437
- 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 421
- 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
- 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
- 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
- 2023An Embodied Generalist Agent in 3D Worldcited by 355
- 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
- 2016Virtual Worlds as Proxy for Multi-Object Tracking Analysiscited by 340
- 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 340
- 2018ABC: A Big CAD Model Dataset For Geometric Deep Learningcited by 339
50 most cited of 96 All 96 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Bridging the Dimensionality Gap: A Taxonomy and Survey of 2D Vision Model Adaptation for 3D Analysis
- 2026Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective
- 2024Diffusion Models in 3D Vision: A Survey
- 2022Deep Generative Models on 3D Representations: A Survey
- 2024ImageNet3D: Towards General-Purpose Object-Level 3D Understanding
- 2025Geometry and Perception Guided Gaussians for Multiview-consistent 3D Generation from a Single Image
- 2023Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Models
- 20263D-IDE: 3D Implicit Depth Emergent