# Novel View Synthesis with View-Dependent Effects from a Single Image

arXiv: https://arxiv.org/abs/2312.08071 · HTML: https://mlsanity.com/paper/2312.08071/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 50 references that are in the map.
- First posted on arXiv 13 Dec 2023.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures](https://mlsanity.com/paper/2208.00277/) (2022)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [DynIBaR: Neural Dynamic Image-Based Rendering](https://mlsanity.com/paper/2211.11082/) (2022)
- [NeX: Real-time View Synthesis with Neural Basis Expansion](https://mlsanity.com/paper/2103.05606/) (2021)
- [MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis](https://mlsanity.com/paper/2103.14910/) (2021)
- [3D Ken Burns Effect from a Single Image](https://mlsanity.com/paper/1909.05483/) (2019)
- [Learning the Depths of Moving People by Watching Frozen People](https://mlsanity.com/paper/1904.11111/) (2019)
- [Consistent View Synthesis with Pose-Guided Diffusion Models](https://mlsanity.com/paper/2303.17598/) (2023)
- [Vision Transformer for NeRF-Based View Synthesis from a Single Input Image](https://mlsanity.com/paper/2207.05736/) (2022)

## Built on by (most cited first, 2 of 2)

- [SpatialDreamer: Self-supervised Stereo Video Synthesis from Monocular Input](https://mlsanity.com/paper/2411.11934/) (2024)
- [PanoDreamer: Optimization-Based Single Image to 360 3D Scene With Diffusion](https://mlsanity.com/paper/2412.04827/) (2024)

## Abstract (arXiv)

In this paper, we firstly consider view-dependent effects into single image-based novel view synthesis (NVS) problems. For this, we propose to exploit the camera motion priors in NVS to model view-dependent appearance or effects (VDE) as the negative disparity in the scene. By recognizing specularities "follow" the camera motion, we infuse VDEs into the input images by aggregating input pixel colors along the negative depth region of the epipolar lines. Also, we propose a `relaxed volumetric rendering' approximation that allows computing the densities in a single pass, improving efficiency for NVS from single images. Our method can learn single-image NVS from image sequences only, which is a completely self-supervised learning method, for the first time requiring neither depth nor camera pose annotations. We present extensive experiment results and show that our proposed method can learn NVS with VDEs, outperforming the SOTA single-view NVS methods on the RealEstate10k and MannequinChallenge datasets.