# Patch-based 3D Natural Scene Generation from a Single Example

arXiv: https://arxiv.org/abs/2304.12670 · HTML: https://mlsanity.com/paper/2304.12670/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 110 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 30 of 45)

- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis](https://mlsanity.com/paper/2110.08985/) (2021)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [Free View Synthesis](https://mlsanity.com/paper/2008.05511/) (2020)
- [Stable View Synthesis](https://mlsanity.com/paper/2011.07233/) (2020)
- [Unconstrained Scene Generation with Locally Conditioned Radiance Fields](https://mlsanity.com/paper/2104.00670/) (2021)
- [GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds](https://mlsanity.com/paper/2104.07659/) (2021)
- [Improved Techniques for Training Single-Image GANs](https://mlsanity.com/paper/2003.11512/) (2020)
- [Positional Encoding as Spatial Inductive Bias in GANs](https://mlsanity.com/paper/2012.05217/) (2020)
- [Deep Generative Modeling for Scene Synthesis via Hybrid Representations](https://mlsanity.com/paper/1808.02084/) (2018)

## Built on by (most cited first, 15 of 15)

- [Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape](https://mlsanity.com/paper/2305.15399/) (2023)
- [NuiScene: Exploring Efficient Generation of Unbounded Outdoor Scenes](https://mlsanity.com/paper/2503.16375/) (2025)
- [ThemeStation: Generating Theme-Aware 3D Assets from Few Exemplars](https://mlsanity.com/paper/2403.15383/) (2024)
- [ShaDDR: Interactive Example-Based Geometry and Texture Generation via 3D Shape Detailization and Differentiable Rendering](https://mlsanity.com/paper/2306.04889/) (2023)
- [DECOLLAGE: 3D Detailization by Controllable, Localized, and Learned Geometry Enhancement](https://mlsanity.com/paper/2409.06129/) (2024)
- [PEGAsus: 3D Personalization of Geometry and Appearance](https://mlsanity.com/paper/2602.08198/) (2026)
- [GaussianSeal: Rooting Adaptive Watermarks for 3D Gaussian Generation Model](https://mlsanity.com/paper/2503.00531/) (2025)
- [Towards Realistic Example-based Modeling via 3D Gaussian Stitching](https://mlsanity.com/paper/2408.15708/) (2024)
- [ShapeShifter: 3D Variations Using Multiscale and Sparse Point-Voxel Diffusion](https://mlsanity.com/paper/2502.02187/) (2025)
- [Category-Aware 3D Object Composition with Disentangled Texture and Shape Multi-view Diffusion](https://mlsanity.com/paper/2509.02357/) (2025)
- [NeRF synthesis with shading guidance](https://mlsanity.com/paper/2306.11556/) (2023)
- [GOEmbed: Gradient Origin Embeddings for Representation Agnostic 3D Feature Learning](https://mlsanity.com/paper/2312.08744/) (2023)
- [RoamScene3D: Immersive Text-to-3D Scene Generation via Adaptive Object-aware Roaming](https://mlsanity.com/paper/2601.19433/) (2026)
- [ExCellGen: Fast, Controllable, Photorealistic 3D Scene Generation from a Single Real-World Exemplar](https://mlsanity.com/paper/2412.16253/) (2024)
- [3D Shape Generation: A Survey](https://mlsanity.com/paper/2506.22678/) (2025)

## Abstract (arXiv)

We target a 3D generative model for general natural scenes that are typically unique and intricate. Lacking the necessary volumes of training data, along with the difficulties of having ad hoc designs in presence of varying scene characteristics, renders existing setups intractable. Inspired by classical patch-based image models, we advocate for synthesizing 3D scenes at the patch level, given a single example. At the core of this work lies important algorithmic designs w.r.t the scene representation and generative patch nearest-neighbor module, that address unique challenges arising from lifting classical 2D patch-based framework to 3D generation. These design choices, on a collective level, contribute to a robust, effective, and efficient model that can generate high-quality general natural scenes with both realistic geometric structure and visual appearance, in large quantities and varieties, as demonstrated upon a variety of exemplar scenes.