Novel View Synthesis with View-Dependent Effects from a Single Image
Not on the map. Why?
Abstract (arXiv)
In this paper, we firstly consider view-dependent effects into single image-based novel view synthesis (NVS) problems. For this, we propose to exploit the camera motion priors in NVS to model view-dependent appearance or effects (VDE) as the negative disparity in the scene. By recognizing specularities "follow" the camera motion, we infuse VDEs into the input images by aggregating input pixel colors along the negative depth region of the epipolar lines. Also, we propose a `relaxed volumetric rendering' approximation that allows computing the densities in a single pass, improving efficiency for NVS from single images. Our method can learn single-image NVS from image sequences only, which is a completely self-supervised learning method, for the first time requiring neither depth nor camera pose annotations. We present extensive experiment results and show that our proposed method can learn NVS with VDEs, outperforming the SOTA single-view NVS methods on the RealEstate10k and MannequinChallenge datasets.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 13 Dec 2023.
Builds on
41 of 50 references.
References · 41
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
- 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
- 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
- 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
- 2021Vision Transformers for Dense Predictioncited by 1,803
29 more
- 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,613
- 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
- 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,205
- 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
- 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 905
- 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 820
- 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 782
- 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 723
- 2019Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelinescited by 672
- 2022MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architecturescited by 311
- 2020Single-View View Synthesis with Multiplane Imagescited by 263
- 2022DynIBaR: Neural Dynamic Image-Based Renderingcited by 224
- 2021NeX: Real-time View Synthesis with Neural Basis Expansioncited by 216
- 2021MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesiscited by 133
- 20193D Ken Burns Effect from a Single Imagecited by 128
- 2019Learning the Depths of Moving People by Watching Frozen Peoplecited by 120
- 2023Consistent View Synthesis with Pose-Guided Diffusion Modelscited by 117
- 2022Vision Transformer for NeRF-Based View Synthesis from a Single Input Imagecited by 104
- 2021Geometry-Free View Synthesis: Transformers and no 3D Priorscited by 100
- 2016Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networkscited by 73
- 2023Behind the Scenes: Density Fields for Single View Reconstructioncited by 63
- 2022Single-View View Synthesis in the Wild with Learned Adaptive Multiplane Imagescited by 59
- 2022Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Imagecited by 55
- 2020Forget About the LiDAR: Self-Supervised Depth Estimators with MED Probability Volumescited by 50
- 2022SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fieldscited by 49
- 2020One Shot 3D Photographycited by 45
- 2023VQ3D: Learning a 3D-Aware Generative Model on ImageNetcited by 32
- 2021PLADE-Net: Towards Pixel-Level Accuracy for Self-Supervised Single-View Depth Estimation with Neural Positional Encoding and Distilled Matting Losscited by 31
- 2022DevNet: Self-supervised Monocular Depth Learning via Density Volume Constructioncited by 22
Built on by
Citing papers per year: 2024: 2.
2 papers.