# Exploiting Priors from 3D Diffusion Models for RGB-Based One-Shot View Planning

arXiv: https://arxiv.org/abs/2403.16803 · HTML: https://mlsanity.com/paper/2403.16803/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 25 Mar 2024.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 25 of 25)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion](https://mlsanity.com/paper/2311.07885/) (2023)
- [SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views](https://mlsanity.com/paper/2206.05737/) (2022)
- [ActiveNeRF: Learning where to See with Uncertainty Estimation](https://mlsanity.com/paper/2209.08546/) (2022)
- [HomebrewedDB: RGB-D Dataset for 6D Pose Estimation of 3D Objects](https://mlsanity.com/paper/1904.03167/) (2019)
- [Density-aware NeRF Ensembles: Quantifying Predictive Uncertainty in Neural Radiance Fields](https://mlsanity.com/paper/2209.08718/) (2022)
- [NeU-NBV: Next Best View Planning Using Uncertainty Estimation in Image-Based Neural Rendering](https://mlsanity.com/paper/2303.01284/) (2023)
- [Supervised Learning of the Next-Best-View for 3D Object Reconstruction](https://mlsanity.com/paper/1905.05833/) (2019)
- [FvOR: Robust Joint Shape and Pose Optimization for Few-view Object Reconstruction](https://mlsanity.com/paper/2205.07763/) (2022)
- [NBV-SC: Next Best View Planning based on Shape Completion for Fruit Mapping and Reconstruction](https://mlsanity.com/paper/2209.15376/) (2022)
- [How Many Views Are Needed to Reconstruct an Unknown Object Using NeRF?](https://mlsanity.com/paper/2310.00684/) (2023)
- [Viewpoint Push Planning for Mapping of Unknown Confined Spaces](https://mlsanity.com/paper/2303.03126/) (2023)
- [Graph-based View Motion Planning for Fruit Detection](https://mlsanity.com/paper/2303.03048/) (2023)
- [Integrating One-Shot View Planning with a Single Next-Best View via Long-Tail Multiview Sampling](https://mlsanity.com/paper/2304.00910/) (2023)
- [Active Implicit Reconstruction Using One-Shot View Planning](https://mlsanity.com/paper/2310.00685/) (2023)

## Built on by (most cited first, 4 of 4)

- [Diffusion Models for Robotic Manipulation: A Survey](https://mlsanity.com/paper/2504.08438/) (2025)
- [Diffusion Model for Planning: A Systematic Literature Review](https://mlsanity.com/paper/2408.10266/) (2024)
- [DM-OSVP++: One-Shot View Planning Using 3D Diffusion Models for Active RGB-Based Object Reconstruction](https://mlsanity.com/paper/2504.11674/) (2025)
- [ObjSplat: Geometry-Aware Gaussian Surfels for Active Object Reconstruction](https://mlsanity.com/paper/2601.06997/) (2026)

## Abstract (arXiv)

Object reconstruction is relevant for many autonomous robotic tasks that require interaction with the environment. A key challenge in such scenarios is planning view configurations to collect informative measurements for reconstructing an initially unknown object. One-shot view planning enables efficient data collection by predicting view configurations and planning the globally shortest path connecting all views at once. However, prior knowledge about the object is required to conduct one-shot view planning. In this work, we propose a novel one-shot view planning approach that utilizes the powerful 3D generation capabilities of diffusion models as priors. By incorporating such geometric priors into our pipeline, we achieve effective one-shot view planning starting with only a single RGB image of the object to be reconstructed. Our planning experiments in simulation and real-world setups indicate that our approach balances well between object reconstruction quality and movement cost.