# Stylizing 3D Scene via Implicit Representation and HyperNetwork

arXiv: https://arxiv.org/abs/2105.13016 · HTML: https://mlsanity.com/paper/2105.13016/

## Key facts

- Cited by 103 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 50 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 30 of 31)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Universal Style Transfer via Feature Transforms](https://mlsanity.com/paper/1705.08086/) (2017)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [Free View Synthesis](https://mlsanity.com/paper/2008.05511/) (2020)
- [Texture Networks: Feed-forward Synthesis of Textures and Stylized Images](https://mlsanity.com/paper/1603.03417/) (2016)
- [Stable View Synthesis](https://mlsanity.com/paper/2011.07233/) (2020)
- [Demystifying Neural Style Transfer](https://mlsanity.com/paper/1701.01036/) (2017)
- [Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction](https://mlsanity.com/paper/1706.07036/) (2017)
- [NeuTex: Neural Texture Mapping for Volumetric Neural Rendering](https://mlsanity.com/paper/2103.00762/) (2021)
- [Coherent Online Video Style Transfer](https://mlsanity.com/paper/1703.09211/) (2017)
- [Arbitrary Video Style Transfer via Multi-Channel Correlation](https://mlsanity.com/paper/2009.08003/) (2020)
- [Diversified Texture Synthesis with Feed-forward Networks](https://mlsanity.com/paper/1703.01664/) (2017)
- [Learning Linear Transformations for Fast Arbitrary Style Transfer](https://mlsanity.com/paper/1808.04537/) (2018)
- [Learning single-image 3D reconstruction by generative modelling of shape, pose and shading](https://mlsanity.com/paper/1901.06447/) (2019)
- [Two-Stage Peer-Regularized Feature Recombination for Arbitrary Image Style Transfer](https://mlsanity.com/paper/1906.02913/) (2019)
- [Optical Flow Distillation: Towards Efficient and Stable Video Style Transfer](https://mlsanity.com/paper/2007.05146/) (2020)
- [Characterizing and Improving Stability in Neural Style Transfer](https://mlsanity.com/paper/1705.02092/) (2017)
- [Image2Mesh: A Learning Framework for Single Image 3D Reconstruction](https://mlsanity.com/paper/1711.10669/) (2017)
- [ReCoNet: Real-time Coherent Video Style Transfer Network](https://mlsanity.com/paper/1807.01197/) (2018)

## Built on by (most cited first, 30 of 103)

- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions](https://mlsanity.com/paper/2303.12789/) (2023)
- [GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions](https://mlsanity.com/paper/2311.16037/) (2023)
- [ARF: Artistic Radiance Fields](https://mlsanity.com/paper/2206.06360/) (2022)
- [StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning](https://mlsanity.com/paper/2205.12183/) (2022)
- [NeRF-Art: Text-Driven Neural Radiance Fields Stylization](https://mlsanity.com/paper/2212.08070/) (2022)
- [SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field](https://mlsanity.com/paper/2303.13277/) (2023)
- [DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editing](https://mlsanity.com/paper/2404.18929/) (2024)
- [AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control](https://mlsanity.com/paper/2303.17606/) (2023)
- [StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields](https://mlsanity.com/paper/2303.10598/) (2023)
- [StyleGaussian: Instant 3D Style Transfer with Gaussian Splatting](https://mlsanity.com/paper/2403.07807/) (2024)
- [FocalDreamer: Text-driven 3D Editing via Focal-fusion Assembly](https://mlsanity.com/paper/2308.10608/) (2023)
- [ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields](https://mlsanity.com/paper/2402.00864/) (2024)
- [Unified Implicit Neural Stylization](https://mlsanity.com/paper/2204.01943/) (2022)
- [StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions](https://mlsanity.com/paper/2112.01530/) (2021)
- [UPST-NeRF: Universal Photorealistic Style Transfer of Neural Radiance Fields for 3D Scene](https://mlsanity.com/paper/2208.07059/) (2022)
- [3D Photo Stylization: Learning to Generate Stylized Novel Views from a Single Image](https://mlsanity.com/paper/2112.00169/) (2021)
- [SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priors](https://mlsanity.com/paper/2311.17261/) (2023)
- [ClimateNeRF: Extreme Weather Synthesis in Neural Radiance Field](https://mlsanity.com/paper/2211.13226/) (2022)
- [Ref-NPR: Reference-Based Non-Photorealistic Radiance Fields for Controllable Scene Stylization](https://mlsanity.com/paper/2212.02766/) (2022)
- [CopyRNeRF: Protecting the CopyRight of Neural Radiance Fields](https://mlsanity.com/paper/2307.11526/) (2023)
- [SNeRF: Stylized Neural Implicit Representations for 3D Scenes](https://mlsanity.com/paper/2207.02363/) (2022)
- [DreamSpace: Dreaming Your Room Space with Text-Driven Panoramic Texture Propagation](https://mlsanity.com/paper/2310.13119/) (2023)
- [Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields](https://mlsanity.com/paper/2306.12760/) (2023)
- [StylizedGS: Controllable Stylization for 3D Gaussian Splatting](https://mlsanity.com/paper/2404.05220/) (2024)
- [Locally Stylized Neural Radiance Fields](https://mlsanity.com/paper/2309.10684/) (2023)
- [Taming Latent Diffusion Model for Neural Radiance Field Inpainting](https://mlsanity.com/paper/2404.09995/) (2024)
- [SGSST: Scaling Gaussian Splatting StyleTransfer](https://mlsanity.com/paper/2412.03371/) (2024)
- [ProteusNeRF: Fast Lightweight NeRF Editing using 3D-Aware Image Context](https://mlsanity.com/paper/2310.09965/) (2023)
- [Neural Parameterization for Dynamic Human Head Editing](https://mlsanity.com/paper/2207.00210/) (2022)

## Abstract (arXiv)

In this work, we aim to address the 3D scene stylization problem - generating stylized images of the scene at arbitrary novel view angles. A straightforward solution is to combine existing novel view synthesis and image/video style transfer approaches, which often leads to blurry results or inconsistent appearance. Inspired by the high-quality results of the neural radiance fields (NeRF) method, we propose a joint framework to directly render novel views with the desired style. Our framework consists of two components: an implicit representation of the 3D scene with the neural radiance fields model, and a hypernetwork to transfer the style information into the scene representation. In particular, our implicit representation model disentangles the scene into the geometry and appearance branches, and the hypernetwork learns to predict the parameters of the appearance branch from the reference style image. To alleviate the training difficulties and memory burden, we propose a two-stage training procedure and a patch sub-sampling approach to optimize the style and content losses with the neural radiance fields model. After optimization, our model is able to render consistent novel views at arbitrary view angles with arbitrary style. Both quantitative evaluation and human subject study have demonstrated that the proposed method generates faithful stylization results with consistent appearance across different views.