# Factorized and Controllable Neural Re-Rendering of Outdoor Scene for Photo Extrapolation

arXiv: https://arxiv.org/abs/2207.06899 · HTML: https://mlsanity.com/paper/2207.06899/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 66 references that are in the map.
- First posted on arXiv 14 Jul 2022.

## Builds on (most cited first, 30 of 36)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [Learning Continuous Image Representation with Local Implicit Image Function](https://mlsanity.com/paper/2012.09161/) (2020)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering](https://mlsanity.com/paper/2109.01847/) (2021)
- [Image Matching across Wide Baselines: From Paper to Practice](https://mlsanity.com/paper/2003.01587/) (2020)
- [3D Photography using Context-aware Layered Depth Inpainting](https://mlsanity.com/paper/2004.04727/) (2020)
- [Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition](https://mlsanity.com/paper/2110.14373/) (2021)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination](https://mlsanity.com/paper/2106.01970/) (2021)
- [Hallucinated Neural Radiance Fields in the Wild](https://mlsanity.com/paper/2111.15246/) (2021)
- [3D Ken Burns Effect from a Single Image](https://mlsanity.com/paper/1909.05483/) (2019)
- [Neural Rerendering in the Wild](https://mlsanity.com/paper/1904.04290/) (2019)
- [Multi-Scale Geometric Consistency Guided Multi-View Stereo](https://mlsanity.com/paper/1904.08103/) (2019)
- [GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds](https://mlsanity.com/paper/2104.07659/) (2021)
- [Object-Centric Neural Scene Rendering](https://mlsanity.com/paper/2012.08503/) (2020)
- [CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering](https://mlsanity.com/paper/1808.08601/) (2018)
- [PixelSynth: Generating a 3D-Consistent Experience from a Single Image](https://mlsanity.com/paper/2108.05892/) (2021)
- [Deep Sky Modeling for Single Image Outdoor Lighting Estimation](https://mlsanity.com/paper/1905.03897/) (2019)
- [Boundless: Generative Adversarial Networks for Image Extension](https://mlsanity.com/paper/1908.07007/) (2019)
- [Crowdsampling the Plenoptic Function](https://mlsanity.com/paper/2007.15194/) (2020)
- [Very Long Natural Scenery Image Prediction by Outpainting](https://mlsanity.com/paper/1912.12688/) (2019)
- [Self-supervised Outdoor Scene Relighting](https://mlsanity.com/paper/2107.03106/) (2021)

## Built on by (most cited first, 16 of 16)

- [NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images](https://mlsanity.com/paper/2305.17398/) (2023)
- [NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editing](https://mlsanity.com/paper/2207.11911/) (2022)
- [SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field](https://mlsanity.com/paper/2303.13277/) (2023)
- [Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes](https://mlsanity.com/paper/2304.03266/) (2023)
- [IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis](https://mlsanity.com/paper/2210.00647/) (2022)
- [PNeRFLoc: Visual Localization with Point-based Neural Radiance Fields](https://mlsanity.com/paper/2312.10649/) (2023)
- [Mirror-NeRF: Learning Neural Radiance Fields for Mirrors with Whitted-Style Ray Tracing](https://mlsanity.com/paper/2308.03280/) (2023)
- [Learning Unified Decompositional and Compositional NeRF for Editable Novel View Synthesis](https://mlsanity.com/paper/2308.02840/) (2023)
- [GeneAvatar: Generic Expression-Aware Volumetric Head Avatar Editing from a Single Image](https://mlsanity.com/paper/2404.02152/) (2024)
- [GaussianUpdate: Continual 3D Gaussian Splatting Update for Changing Environments](https://mlsanity.com/paper/2508.08867/) (2025)
- [Neural Scene Chronology](https://mlsanity.com/paper/2306.07970/) (2023)
- [Semantics-Controlled Gaussian Splatting for Outdoor Scene Reconstruction and Rendering in Virtual Reality](https://mlsanity.com/paper/2409.15959/) (2024)
- [Seg-Wild: Interactive Segmentation based on 3D Gaussian Splatting for Unconstrained Image Collections](https://mlsanity.com/paper/2507.07395/) (2025)
- [NeRFs are Mirror Detectors: Using Structural Similarity for Multi-View Mirror Scene Reconstruction with 3D Surface Primitives](https://mlsanity.com/paper/2501.04074/) (2025)
- [WilLaGS: Latent-Conditional 3D Appearance Fields for Robust Gaussian Splatting In-the-Wild](https://mlsanity.com/paper/2608.28240/) (2026)
- [NeuMesh++: Towards Versatile and Efficient Volumetric Editing with Disentangled Neural Mesh-based Implicit Field](https://mlsanity.com/paper/2606.19316/) (2026)

## Abstract (arXiv)

Expanding an existing tourist photo from a partially captured scene to a full scene is one of the desired experiences for photography applications. Although photo extrapolation has been well studied, it is much more challenging to extrapolate a photo (i.e., selfie) from a narrow field of view to a wider one while maintaining a similar visual style. In this paper, we propose a factorized neural re-rendering model to produce photorealistic novel views from cluttered outdoor Internet photo collections, which enables the applications including controllable scene re-rendering, photo extrapolation and even extrapolated 3D photo generation. Specifically, we first develop a novel factorized re-rendering pipeline to handle the ambiguity in the decomposition of geometry, appearance and illumination. We also propose a composited training strategy to tackle the unexpected occlusion in Internet images. Moreover, to enhance photo-realism when extrapolating tourist photographs, we propose a novel realism augmentation process to complement appearance details, which automatically propagates the texture details from a narrow captured photo to the extrapolated neural rendered image. The experiments and photo editing examples on outdoor scenes demonstrate the superior performance of our proposed method in both photo-realism and downstream applications.