Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask
Not on the map. Why?
Abstract (arXiv)
We introduce a novel approach that takes a single semantic mask as input to synthesize multi-view consistent color images of natural scenes, trained with a collection of single images from the Internet. Prior works on 3D-aware image synthesis either require multi-view supervision or learning category-level prior for specific classes of objects, which can hardly work for natural scenes. Our key idea to solve this challenging problem is to use a semantic field as the intermediate representation, which is easier to reconstruct from an input semantic mask and then translate to a radiance field with the assistance of off-the-shelf semantic image synthesis models. Experiments show that our method outperforms baseline methods and produces photorealistic, multi-view consistent videos of a variety of natural scenes.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 72 of its 93 references that are in the map. First posted on arXiv 14 Feb 2023.
Builds on
72 of 93 references.
References · 72
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
- 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
- 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
- 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
- 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
- 2014Conditional Generative Adversarial Netscited by 2,152
- 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
- 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
- 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
- 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
38 more
- 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
- 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
- 2018Demystifying MMD GANscited by 1,268
- 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
- 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
- 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
- 2021Volume Rendering of Neural Implicit Surfacescited by 990
- 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
- 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
- 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
- 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
- 2020Implicit Geometric Regularization for Learning Shapescited by 707
- 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
- 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 643
- 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 614
- 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
- 2022MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstructioncited by 467
- 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
- 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
- 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
- 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 361
- 2020Virtual KITTI 2cited by 345
- 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
- 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
- 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
- 2020Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Imagecited by 239
- 2021DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networkscited by 232
- 2018Stereo Magnification: Learning View Synthesis using Multiplane Imagescited by 206
- 20203D Photography using Context-aware Layered Depth Inpaintingcited by 206
- 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
- 2016Single-Image Depth Perception in the Wildcited by 179
- 2022SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Imagecited by 165
- 2022Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentationcited by 152
- 2022ARF: Artistic Radiance Fieldscited by 147
- 2021Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesiscited by 140
- 2022StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learningcited by 140
- 2021MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesiscited by 133
- 2022VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Gridscited by 133
50 most cited of 72 All 72 (JSON)
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.