# Semantic View Synthesis

arXiv: https://arxiv.org/abs/2008.10598 · HTML: https://mlsanity.com/paper/2008.10598/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Aug 2020.

## Built on by (most cited first, 18 of 18)

- [StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning](https://mlsanity.com/paper/2205.12183/) (2022)
- [Consistent View Synthesis with Pose-Guided Diffusion Models](https://mlsanity.com/paper/2303.17598/) (2023)
- [Regularizing Generative Adversarial Networks under Limited Data](https://mlsanity.com/paper/2104.03310/) (2021)
- [Neural Trajectory Fields for Dynamic Novel View Synthesis](https://mlsanity.com/paper/2105.05994/) (2021)
- [Learning to Stylize Novel Views](https://mlsanity.com/paper/2105.13509/) (2021)
- [UPST-NeRF: Universal Photorealistic Style Transfer of Neural Radiance Fields for 3D Scene](https://mlsanity.com/paper/2208.07059/) (2022)
- [Neural Deformable Voxel Grid for Fast Optimization of Dynamic View Synthesis](https://mlsanity.com/paper/2206.07698/) (2022)
- [Guided Co-Modulated GAN for 360deg Field of View Extrapolation](https://mlsanity.com/paper/2204.07286/) (2022)
- [Sem2NeRF: Converting Single-View Semantic Masks to Neural Radiance Fields](https://mlsanity.com/paper/2203.10821/) (2022)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [3D-aware Conditional Image Synthesis](https://mlsanity.com/paper/2302.08509/) (2023)
- [Hybrid Neural Fusion for Full-frame Video Stabilization](https://mlsanity.com/paper/2102.06205/) (2021)
- [Vector Quantized Image-to-Image Translation](https://mlsanity.com/paper/2207.13286/) (2022)
- [Embedding Novel Views in a Single JPEG Image](https://mlsanity.com/paper/2108.13003/) (2021)
- [Context-Consistent Semantic Image Editing with Style-Preserved Modulation](https://mlsanity.com/paper/2207.06252/) (2022)
- [A Shared Representation for Photorealistic Driving Simulators](https://mlsanity.com/paper/2112.05134/) (2021)
- [Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask](https://mlsanity.com/paper/2302.07224/) (2023)
- [Learning multiplane images from single views with self-supervision](https://mlsanity.com/paper/2110.09380/) (2021)

## Abstract (arXiv)

We tackle a new problem of semantic view synthesis -- generating free-viewpoint rendering of a synthesized scene using a semantic label map as input. We build upon recent advances in semantic image synthesis and view synthesis for handling photographic image content generation and view extrapolation. Direct application of existing image/view synthesis methods, however, results in severe ghosting/blurry artifacts. To address the drawbacks, we propose a two-step approach. First, we focus on synthesizing the color and depth of the visible surface of the 3D scene. We then use the synthesized color and depth to impose explicit constraints on the multiple-plane image (MPI) representation prediction process. Our method produces sharp contents at the original view and geometrically consistent renderings across novel viewpoints. The experiments on numerous indoor and outdoor images show favorable results against several strong baselines and validate the effectiveness of our approach.