# UPST-NeRF: Universal Photorealistic Style Transfer of Neural Radiance Fields for 3D Scene

arXiv: https://arxiv.org/abs/2208.07059 · HTML: https://mlsanity.com/paper/2208.07059/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 70 references that are in the map.
- First posted on arXiv 15 Aug 2022.

## Builds on (most cited first, 30 of 44)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://mlsanity.com/paper/1812.01024/) (2018)
- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [DeepView: View Synthesis with Learned Gradient Descent](https://mlsanity.com/paper/1906.07316/) (2019)
- [NeX: Real-time View Synthesis with Neural Basis Expansion](https://mlsanity.com/paper/2103.05606/) (2021)
- [Pushing the Boundaries of View Extrapolation with Multiplane Images](https://mlsanity.com/paper/1905.00413/) (2019)
- [Deep Photo Style Transfer](https://mlsanity.com/paper/1703.07511/) (2017)
- [Photorealistic Style Transfer via Wavelet Transforms](https://mlsanity.com/paper/1903.09760/) (2019)
- [ARF: Artistic Radiance Fields](https://mlsanity.com/paper/2206.06360/) (2022)
- [StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning](https://mlsanity.com/paper/2205.12183/) (2022)
- [A Closed-form Solution to Photorealistic Image Stylization](https://mlsanity.com/paper/1802.06474/) (2018)
- [3D Ken Burns Effect from a Single Image](https://mlsanity.com/paper/1909.05483/) (2019)
- [Neural Rerendering in the Wild](https://mlsanity.com/paper/1904.04290/) (2019)

## Built on by (most cited first, 30 of 47)

- [SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field](https://mlsanity.com/paper/2303.13277/) (2023)
- [StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields](https://mlsanity.com/paper/2303.10598/) (2023)
- [FocalDreamer: Text-driven 3D Editing via Focal-fusion Assembly](https://mlsanity.com/paper/2308.10608/) (2023)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fields](https://mlsanity.com/paper/2212.10699/) (2022)
- [SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priors](https://mlsanity.com/paper/2311.17261/) (2023)
- [ClimateNeRF: Extreme Weather Synthesis in Neural Radiance Field](https://mlsanity.com/paper/2211.13226/) (2022)
- [RecolorNeRF: Layer Decomposed Radiance Fields for Efficient Color Editing of 3D Scenes](https://mlsanity.com/paper/2301.07958/) (2023)
- [DreamSpace: Dreaming Your Room Space with Text-Driven Panoramic Texture Propagation](https://mlsanity.com/paper/2310.13119/) (2023)
- [Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields](https://mlsanity.com/paper/2306.12760/) (2023)
- [Super-NeRF: View-consistent Detail Generation for NeRF super-resolution](https://mlsanity.com/paper/2304.13518/) (2023)
- [Gaussian Splatting in Style](https://mlsanity.com/paper/2403.08498/) (2024)
- [NeRFs in Robotics: A Survey](https://mlsanity.com/paper/2405.01333/) (2024)
- [Transforming Radiance Field with Lipschitz Network for Photorealistic 3D Scene Stylization](https://mlsanity.com/paper/2303.13232/) (2023)
- [CoARF: Controllable 3D Artistic Style Transfer for Radiance Fields](https://mlsanity.com/paper/2404.14967/) (2024)
- [NLUT: Neural-based 3D Lookup Tables for Video Photorealistic Style Transfer](https://mlsanity.com/paper/2303.09170/) (2023)
- [FPRF: Feed-Forward Photorealistic Style Transfer of Large-Scale 3D Neural Radiance Fields](https://mlsanity.com/paper/2401.05516/) (2024)
- [Editing 3D Scenes via Text Prompts without Retraining](https://mlsanity.com/paper/2309.04917/) (2023)
- [Style3D: Attention-guided Multi-view Style Transfer for 3D Object Generation](https://mlsanity.com/paper/2412.03571/) (2024)
- [Interactive Geometry Editing of Neural Radiance Fields](https://mlsanity.com/paper/2303.11537/) (2023)
- [ARF-Plus: Controlling Perceptual Factors in Artistic Radiance Fields for 3D Scene Stylization](https://mlsanity.com/paper/2308.12452/) (2023)
- [HyperLips: Hyper Control Lips with High Resolution Decoder for Talking Face Generation](https://mlsanity.com/paper/2310.05720/) (2023)
- [ConRF: Zero-shot Stylization of 3D Scenes with Conditioned Radiation Fields](https://mlsanity.com/paper/2402.01950/) (2024)
- [SeamlessNeRF: Stitching Part NeRFs with Gradient Propagation](https://mlsanity.com/paper/2311.16127/) (2023)
- [3DStyleGLIP: Part-Tailored Text-Guided 3D Neural Stylization](https://mlsanity.com/paper/2404.02634/) (2024)
- [Instant Photorealistic Neural Radiance Fields Stylization](https://mlsanity.com/paper/2303.16884/) (2023)
- [3D Face Style Transfer with a Hybrid Solution of NeRF and Mesh Rasterization](https://mlsanity.com/paper/2311.13168/) (2023)
- [SplatMesh: Interactive 3D Segmentation and Editing Using Mesh-Based Gaussian Splatting](https://mlsanity.com/paper/2312.15856/) (2023)
- [S-DyRF: Reference-Based Stylized Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2403.06205/) (2024)
- [4DStyleGaussian: Zero-shot 4D Style Transfer with Gaussian Splatting](https://mlsanity.com/paper/2410.10412/) (2024)

## Abstract (arXiv)

3D scenes photorealistic stylization aims to generate photorealistic images from arbitrary novel views according to a given style image while ensuring consistency when rendering from different viewpoints. Some existing stylization methods with neural radiance fields can effectively predict stylized scenes by combining the features of the style image with multi-view images to train 3D scenes. However, these methods generate novel view images that contain objectionable artifacts. Besides, they cannot achieve universal photorealistic stylization for a 3D scene. Therefore, a styling image must retrain a 3D scene representation network based on a neural radiation field. We propose a novel 3D scene photorealistic style transfer framework to address these issues. It can realize photorealistic 3D scene style transfer with a 2D style image. We first pre-trained a 2D photorealistic style transfer network, which can meet the photorealistic style transfer between any given content image and style image. Then, we use voxel features to optimize a 3D scene and get the geometric representation of the scene. Finally, we jointly optimize a hyper network to realize the scene photorealistic style transfer of arbitrary style images. In the transfer stage, we use a pre-trained 2D photorealistic network to constrain the photorealistic style of different views and different style images in the 3D scene. The experimental results show that our method not only realizes the 3D photorealistic style transfer of arbitrary style images but also outperforms the existing methods in terms of visual quality and consistency. Project page:https://semchan.github.io/UPST_NeRF.