# Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assembly

arXiv: https://arxiv.org/abs/2410.15971 · HTML: https://mlsanity.com/paper/2410.15971/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 79 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 30 of 62)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction](https://mlsanity.com/paper/2206.00665/) (2022)

## Built on by (most cited first, 30 of 35)

- [MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation](https://mlsanity.com/paper/2412.03558/) (2024)
- [PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers](https://mlsanity.com/paper/2506.05573/) (2025)
- [Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation](https://mlsanity.com/paper/2505.02836/) (2025)
- [DiffGS: Functional Gaussian Splatting Diffusion](https://mlsanity.com/paper/2410.19657/) (2024)
- [Binocular-Guided 3D Gaussian Splatting with View Consistency for Sparse View Synthesis](https://mlsanity.com/paper/2410.18822/) (2024)
- [AutoPartGen: Autogressive 3D Part Generation and Discovery](https://mlsanity.com/paper/2507.13346/) (2025)
- [DepR: Depth Guided Single-view Scene Reconstruction with Instance-level Diffusion](https://mlsanity.com/paper/2507.22825/) (2025)
- [MonoInstance: Enhancing Monocular Priors via Multi-view Instance Alignment for Neural Rendering and Reconstruction](https://mlsanity.com/paper/2503.18363/) (2025)
- [Diorama: Unleashing Zero-shot Single-view 3D Indoor Scene Modeling](https://mlsanity.com/paper/2411.19492/) (2024)
- [UVGS: Reimagining Unstructured 3D Gaussian Splatting using UV Mapping](https://mlsanity.com/paper/2502.01846/) (2025)
- [Learning Bijective Surface Parameterization for Inferring Signed Distance Functions from Sparse Point Clouds with Grid Deformation](https://mlsanity.com/paper/2503.23670/) (2025)
- [Digital Twin Generation from Visual Data: A Survey](https://mlsanity.com/paper/2504.13159/) (2025)
- [GAP: Gaussianize Any Point Clouds with Text Guidance](https://mlsanity.com/paper/2508.05631/) (2025)
- [Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimization](https://mlsanity.com/paper/2507.14841/) (2025)
- [I-Scene: 3D Instance Models are Implicit Generalizable Spatial Learners](https://mlsanity.com/paper/2512.13683/) (2025)
- [Recent Advances in 3D Object and Scene Generation: A Survey](https://mlsanity.com/paper/2504.11734/) (2025)
- [ZeroScene: A Zero-Shot Framework for 3D Scene Generation from a Single Image and Controllable Texture Editing](https://mlsanity.com/paper/2509.23607/) (2025)
- [MoCA: Mixture-of-Components Attention for Scalable Compositional 3D Generation](https://mlsanity.com/paper/2512.07628/) (2025)
- [U-CAN: Unsupervised Point Cloud Denoising with Consistency-Aware Noise2Noise Matching](https://mlsanity.com/paper/2510.25210/) (2025)
- [PixARMesh: Autoregressive Mesh-Native Single-View Scene Reconstruction](https://mlsanity.com/paper/2603.05888/) (2026)
- [AdaSFormer: Adaptive Serialized Transformers for Monocular Semantic Scene Completion from Indoor Environments](https://mlsanity.com/paper/2603.25494/) (2026)
- [φ-Scene: Physically Grounded Image-to-3D Scene Reconstruction](https://mlsanity.com/paper/2606.21596/) (2026)
- [DreamAnywhere: Object-Centric Panoramic 3D Scene Generation](https://mlsanity.com/paper/2506.20367/) (2025)
- [UnPose: Uncertainty-Guided Diffusion Priors for Zero-Shot Pose Estimation](https://mlsanity.com/paper/2508.15972/) (2025)
- [Seeing Through Clutter: Structured 3D Scene Reconstruction via Iterative Object Removal](https://mlsanity.com/paper/2602.04053/) (2026)
- [JRM: Joint Reconstruction Model for Multiple Objects without Alignment](https://mlsanity.com/paper/2603.25985/) (2026)
- [Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling](https://mlsanity.com/paper/2608.30821/) (2026)
- [DM-CFO: A Diffusion Model for Compositional 3D Tooth Generation with Collision-Free Optimization](https://mlsanity.com/paper/2603.03602/) (2026)
- [SGAD-SLAM: Splatting Gaussians at Adjusted Depth for Better Radiance Fields in RGBD SLAM](https://mlsanity.com/paper/2603.21055/) (2026)
- [Pose-Aware Diffusion for 3D Generation](https://mlsanity.com/paper/2605.00345/) (2026)

## Abstract (arXiv)

Large language and vision models have been leading a revolution in visual computing. By greatly scaling up sizes of data and model parameters, the large models learn deep priors which lead to remarkable performance in various tasks. In this work, we present deep prior assembly, a novel framework that assembles diverse deep priors from large models for scene reconstruction from single images in a zero-shot manner. We show that this challenging task can be done without extra knowledge but just simply generalizing one deep prior in one sub-task. To this end, we introduce novel methods related to poses, scales, and occlusion parsing which are keys to enable deep priors to work together in a robust way. Deep prior assembly does not require any 3D or 2D data-driven training in the task and demonstrates superior performance in generalizing priors to open-world scenes. We conduct evaluations on various datasets, and report analysis, numerical and visual comparisons with the latest methods to show our superiority. Project page: https://junshengzhou.github.io/DeepPriorAssembly.