# Deferred Neural Rendering: Image Synthesis using Neural Textures

arXiv: https://arxiv.org/abs/1904.12356 · HTML: https://mlsanity.com/paper/1904.12356/

## Key facts

- Cited by 253 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Apr 2019.

## Built on by (most cited first, 30 of 253)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis](https://mlsanity.com/paper/2312.16812/) (2023)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [Neural RGB-D Surface Reconstruction](https://mlsanity.com/paper/2104.04532/) (2021)
- [MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures](https://mlsanity.com/paper/2208.00277/) (2022)
- [Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain](https://mlsanity.com/paper/2103.01856/) (2021)
- [NeRF-Editing: Geometry Editing of Neural Radiance Fields](https://mlsanity.com/paper/2205.04978/) (2022)
- [NeX: Real-time View Synthesis with Neural Basis Expansion](https://mlsanity.com/paper/2103.05606/) (2021)
- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [Neural Head Avatars from Monocular RGB Videos](https://mlsanity.com/paper/2112.01554/) (2021)
- [WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection](https://mlsanity.com/paper/2101.01456/) (2021)
- [PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering](https://mlsanity.com/paper/2109.08379/) (2021)
- [Neural Rays for Occlusion-aware Image-based Rendering](https://mlsanity.com/paper/2107.13421/) (2021)
- [4D-Rotor Gaussian Splatting: Towards Efficient Novel View Synthesis for Dynamic Scenes](https://mlsanity.com/paper/2402.03307/) (2024)
- [Self-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection](https://mlsanity.com/paper/2203.12208/) (2022)
- [TensoIR: Tensorial Inverse Rendering](https://mlsanity.com/paper/2304.12461/) (2023)
- [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022)
- [Exploring Temporal Coherence for More General Video Face Forgery Detection](https://mlsanity.com/paper/2108.06693/) (2021)
- [Editable Free-viewpoint Video Using a Layered Neural Representation](https://mlsanity.com/paper/2104.14786/) (2021)
- [Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection](https://mlsanity.com/paper/2201.07131/) (2022)
- [Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video](https://mlsanity.com/paper/2210.06108/) (2022)
- [RigNeRF: Fully Controllable Neural 3D Portraits](https://mlsanity.com/paper/2206.06481/) (2022)
- [Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars](https://mlsanity.com/paper/2211.11208/) (2022)
- [Implicit Identity Leakage: The Stumbling Block to Improving Deepfake Detection Generalization](https://mlsanity.com/paper/2210.14457/) (2022)
- [F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories](https://mlsanity.com/paper/2303.15951/) (2023)
- [HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling](https://mlsanity.com/paper/2204.13686/) (2022)
- [M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection](https://mlsanity.com/paper/2104.09770/) (2021)
- [HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs](https://mlsanity.com/paper/2112.02789/) (2021)

## Abstract (arXiv)

The modern computer graphics pipeline can synthesize images at remarkable visual quality; however, it requires well-defined, high-quality 3D content as input. In this work, we explore the use of imperfect 3D content, for instance, obtained from photo-metric reconstructions with noisy and incomplete surface geometry, while still aiming to produce photo-realistic (re-)renderings. To address this challenging problem, we introduce Deferred Neural Rendering, a new paradigm for image synthesis that combines the traditional graphics pipeline with learnable components. Specifically, we propose Neural Textures, which are learned feature maps that are trained as part of the scene capture process. Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline. Both neural textures and deferred neural renderer are trained end-to-end, enabling us to synthesize photo-realistic images even when the original 3D content was imperfect. In contrast to traditional, black-box 2D generative neural networks, our 3D representation gives us explicit control over the generated output, and allows for a wide range of application domains. For instance, we can synthesize temporally-consistent video re-renderings of recorded 3D scenes as our representation is inherently embedded in 3D space. This way, neural textures can be utilized to coherently re-render or manipulate existing video content in both static and dynamic environments at real-time rates. We show the effectiveness of our approach in several experiments on novel view synthesis, scene editing, and facial reenactment, and compare to state-of-the-art approaches that leverage the standard graphics pipeline as well as conventional generative neural networks.