# Neural Rerendering in the Wild

arXiv: https://arxiv.org/abs/1904.04290 · HTML: https://mlsanity.com/paper/1904.04290/

## Key facts

- Cited by 121 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2019.

## Built on by (most cited first, 30 of 121)

- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Point-NeRF: Point-based Neural Radiance Fields](https://mlsanity.com/paper/2201.08845/) (2022)
- [Compact 3D Gaussian Representation for Radiance Field](https://mlsanity.com/paper/2311.13681/) (2023)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose](https://mlsanity.com/paper/2102.06199/) (2021)
- [Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations](https://mlsanity.com/paper/2111.13152/) (2021)
- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction](https://mlsanity.com/paper/2402.17427/) (2024)
- [GeoNeRF: Generalizing NeRF with Geometry Priors](https://mlsanity.com/paper/2111.13539/) (2021)
- [Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes](https://mlsanity.com/paper/2104.06935/) (2021)
- [Shape As Points: A Differentiable Poisson Solver](https://mlsanity.com/paper/2106.03452/) (2021)
- [COTR: Correspondence Transformer for Matching Across Images](https://mlsanity.com/paper/2103.14167/) (2021)
- [GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis](https://mlsanity.com/paper/2312.02155/) (2023)
- [H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion](https://mlsanity.com/paper/2110.13746/) (2021)
- [Editable Free-viewpoint Video Using a Layered Neural Representation](https://mlsanity.com/paper/2104.14786/) (2021)
- [StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning](https://mlsanity.com/paper/2205.12183/) (2022)
- [Hallucinated Neural Radiance Fields in the Wild](https://mlsanity.com/paper/2111.15246/) (2021)
- [Consistent View Synthesis with Pose-Guided Diffusion Models](https://mlsanity.com/paper/2303.17598/) (2023)
- [Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections](https://mlsanity.com/paper/2403.15704/) (2024)
- [NeRF for Outdoor Scene Relighting](https://mlsanity.com/paper/2112.05140/) (2021)
- [Geometry-Free View Synthesis: Transformers and no 3D Priors](https://mlsanity.com/paper/2104.07652/) (2021)
- [Grid-guided Neural Radiance Fields for Large Urban Scenes](https://mlsanity.com/paper/2303.14001/) (2023)
- [T"oRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis](https://mlsanity.com/paper/2109.15271/) (2021)
- [Differentiable Point-Based Radiance Fields for Efficient View Synthesis](https://mlsanity.com/paper/2205.14330/) (2022)

## Abstract (arXiv)

We explore total scene capture -- recording, modeling, and rerendering a scene under varying appearance such as season and time of day. Starting from internet photos of a tourist landmark, we apply traditional 3D reconstruction to register the photos and approximate the scene as a point cloud. For each photo, we render the scene points into a deep framebuffer, and train a neural network to learn the mapping of these initial renderings to the actual photos. This rerendering network also takes as input a latent appearance vector and a semantic mask indicating the location of transient objects like pedestrians. The model is evaluated on several datasets of publicly available images spanning a broad range of illumination conditions. We create short videos demonstrating realistic manipulation of the image viewpoint, appearance, and semantic labeling. We also compare results with prior work on scene reconstruction from internet photos.