# Learning to Stylize Novel Views

arXiv: https://arxiv.org/abs/2105.13509 · HTML: https://mlsanity.com/paper/2105.13509/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- Builds on 86 of its 117 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 30 of 86)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)

## Built on by (most cited first, 30 of 76)

- [Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions](https://mlsanity.com/paper/2303.12789/) (2023)
- [GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions](https://mlsanity.com/paper/2311.16037/) (2023)
- [ARF: Artistic Radiance Fields](https://mlsanity.com/paper/2206.06360/) (2022)
- [StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning](https://mlsanity.com/paper/2205.12183/) (2022)
- [NeRF-Art: Text-Driven Neural Radiance Fields Stylization](https://mlsanity.com/paper/2212.08070/) (2022)
- [DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editing](https://mlsanity.com/paper/2404.18929/) (2024)
- [StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields](https://mlsanity.com/paper/2303.10598/) (2023)
- [StyleGaussian: Instant 3D Style Transfer with Gaussian Splatting](https://mlsanity.com/paper/2403.07807/) (2024)
- [ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields](https://mlsanity.com/paper/2402.00864/) (2024)
- [Unified Implicit Neural Stylization](https://mlsanity.com/paper/2204.01943/) (2022)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions](https://mlsanity.com/paper/2112.01530/) (2021)
- [UPST-NeRF: Universal Photorealistic Style Transfer of Neural Radiance Fields for 3D Scene](https://mlsanity.com/paper/2208.07059/) (2022)
- [3D Photo Stylization: Learning to Generate Stylized Novel Views from a Single Image](https://mlsanity.com/paper/2112.00169/) (2021)
- [Ref-NPR: Reference-Based Non-Photorealistic Radiance Fields for Controllable Scene Stylization](https://mlsanity.com/paper/2212.02766/) (2022)
- [SNeRF: Stylized Neural Implicit Representations for 3D Scenes](https://mlsanity.com/paper/2207.02363/) (2022)
- [StylizedGS: Controllable Stylization for 3D Gaussian Splatting](https://mlsanity.com/paper/2404.05220/) (2024)
- [Super-NeRF: View-consistent Detail Generation for NeRF super-resolution](https://mlsanity.com/paper/2304.13518/) (2023)
- [Locally Stylized Neural Radiance Fields](https://mlsanity.com/paper/2309.10684/) (2023)
- [Chat-Edit-3D: Interactive 3D Scene Editing via Text Prompts](https://mlsanity.com/paper/2407.06842/) (2024)
- [ProteusNeRF: Fast Lightweight NeRF Editing using 3D-Aware Image Context](https://mlsanity.com/paper/2310.09965/) (2023)
- [Efficient-NeRF2NeRF: Streamlining Text-Driven 3D Editing with Multiview Correspondence-Enhanced Diffusion Models](https://mlsanity.com/paper/2312.08563/) (2023)
- [Geometry Transfer for Stylizing Radiance Fields](https://mlsanity.com/paper/2402.00863/) (2024)
- [Gaussian Splatting in Style](https://mlsanity.com/paper/2403.08498/) (2024)
- [Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D](https://mlsanity.com/paper/2403.18922/) (2024)
- [Transforming Radiance Field with Lipschitz Network for Photorealistic 3D Scene Stylization](https://mlsanity.com/paper/2303.13232/) (2023)
- [Reference-based Controllable Scene Stylization with Gaussian Splatting](https://mlsanity.com/paper/2407.07220/) (2024)
- [NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs](https://mlsanity.com/paper/2402.08622/) (2024)
- [InstantStyleGaussian: Efficient Art Style Transfer with 3D Gaussian Splatting](https://mlsanity.com/paper/2408.04249/) (2024)
- [TIGER: Text-Instructed 3D Gaussian Retrieval and Coherent Editing](https://mlsanity.com/paper/2405.14455/) (2024)

## Abstract (arXiv)

We tackle a 3D scene stylization problem - generating stylized images of a scene from arbitrary novel views given a set of images of the same scene and a reference image of the desired style as inputs. Direct solution of combining novel view synthesis and stylization approaches lead to results that are blurry or not consistent across different views. We propose a point cloud-based method for consistent 3D scene stylization. First, we construct the point cloud by back-projecting the image features to the 3D space. Second, we develop point cloud aggregation modules to gather the style information of the 3D scene, and then modulate the features in the point cloud with a linear transformation matrix. Finally, we project the transformed features to 2D space to obtain the novel views. Experimental results on two diverse datasets of real-world scenes validate that our method generates consistent stylized novel view synthesis results against other alternative approaches.