mlsanityData to 7 Oct 2026

The More You See in 2D, the More You Perceive in 3D

arXiv 2404.03652 ↗First posted 4 Apr 2024cs.CVcited by 5

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Humans can infer 3D structure from 2D images of an object based on past experience and improve their 3D understanding as they see more images. Inspired by this behavior, we introduce SAP3D, a system for 3D reconstruction and novel view synthesis from an arbitrary number of unposed images. Given a few unposed images of an object, we adapt a pre-trained view-conditioned diffusion model together with the camera poses of the images via test-time fine-tuning. The adapted diffusion model and the obtained camera poses are then utilized as instance-specific priors for 3D reconstruction and novel view synthesis. We show that as the number of input images increases, the performance of our approach improves, bridging the gap between optimization-based prior-less 3D reconstruction methods and single-image-to-3D diffusion-based methods. We demonstrate our system on real images as well as standard synthetic benchmarks. Our ablation studies confirm that this adaption behavior is key for more accurate 3D understanding.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 45 of its 57 references that are in the map. First posted on arXiv 4 Apr 2024.

Builds on

45 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  2. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization29 Jun
  3. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  4. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  5. 2022Novel View Synthesis with Diffusion Models6 Oct
  6. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  8. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  9. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  12. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
27 more
  1. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  2. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  3. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  4. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  5. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  6. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 614
  7. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  8. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  9. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  10. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  11. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  12. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  13. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  14. 2017Learning a Multi-View Stereo Machinecited by 194
  15. 2023PoseDiffusion: Solving Pose Estimation via Diffusion-aided Bundle Adjustmentcited by 146
  16. 2021NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wildcited by 136
  17. 2020Self-supervised Single-view 3D Reconstruction via Semantic Consistencycited by 127
  18. 2022RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wildcited by 102
  19. 2020Shape and Viewpoint without Keypointscited by 92
  20. 2023RelPose++: Recovering 6D Poses from Sparse-view Observationscited by 87
  21. 2021DOVE: Learning Deformable 3D Objects by Watching Videoscited by 61
  22. 2018Learning to Reconstruct Shapes from Unseen Classescited by 60
  23. 2014Category-Specific Object Reconstruction from a Single Imagecited by 56
  24. 2022SparsePose: Sparse-View Camera Pose Regression and Refinementcited by 53
  25. 2023ViP-NeRF: Visibility Prior for Sparse Input Neural Radiance Fieldscited by 43
  26. 2022Topologically-Aware Deformation Fields for Single-View 3D Reconstructioncited by 29
  27. 2021Differentiable Stereopsis: Meshes from multiple views using differentiable renderingcited by 26

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 2.

5 papers.

  1. 2024Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curationcited by 12
  2. 2025UP2You: Fast Reconstruction of Yourself from Unconstrained Photo Collectionscited by 10
  3. 2025FB-4D: Spatial-Temporal Coherent Dynamic 3D Content Generation with Feature Bankscited by 3
  4. 2024Pragmatist: Multiview Conditional Diffusion Models for High-Fidelity 3D Reconstruction from Unposed Sparse Viewscited by 2
  5. 2024Simple-RF: Regularizing Sparse Input Radiance Fields with Simpler Solutionscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Predicting 3D structure by latent posterior sampling
  2. 2023Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction
  3. 2023ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models
  4. 2024MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for Single or Sparse-view 3D Object Reconstruction
  5. 2023Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views
  6. 2024Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion
  7. 20243D-free meets 3D priors: Novel View Synthesis from a Single Image with Pretrained Diffusion Guidance
  8. 2023iFusion: Inverting Diffusion for Pose-Free Reconstruction from Sparse Views