# FaDIV-Syn: Fast Depth-Independent View Synthesis using Soft Masks and Implicit Blending

arXiv: https://arxiv.org/abs/2106.13139 · HTML: https://mlsanity.com/paper/2106.13139/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 39 of its 58 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 30 of 39)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [DeepMVS: Learning Multi-view Stereopsis](https://mlsanity.com/paper/1804.00650/) (2018)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [Free View Synthesis](https://mlsanity.com/paper/2008.05511/) (2020)
- [Stereo Magnification: Learning View Synthesis using Multiplane Images](https://mlsanity.com/paper/1805.09817/) (2018)
- [3D Photography using Context-aware Layered Depth Inpainting](https://mlsanity.com/paper/2004.04727/) (2020)
- [Pushing the Boundaries of View Extrapolation with Multiplane Images](https://mlsanity.com/paper/1905.00413/) (2019)
- [DeepStereo: Learning to Predict New Views from the World's Imagery](https://mlsanity.com/paper/1506.06825/) (2015)
- [Stable View Synthesis](https://mlsanity.com/paper/2011.07233/) (2020)
- [Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness](https://mlsanity.com/paper/1911.12012/) (2019)
- [Learning-Based View Synthesis for Light Field Cameras](https://mlsanity.com/paper/1609.02974/) (2016)

## Abstract (arXiv)

Novel view synthesis is required in many robotic applications, such as VR teleoperation and scene reconstruction. Existing methods are often too slow for these contexts, cannot handle dynamic scenes, and are limited by their explicit depth estimation stage, where incorrect depth predictions can lead to large projection errors. Our proposed method runs in real time on live streaming data and avoids explicit depth estimation by efficiently warping input images into the target frame for a range of assumed depth planes. The resulting plane sweep volume (PSV) is directly fed into our network, which first estimates soft PSV masks in a self-supervised manner, and then directly produces the novel output view. This improves efficiency and performance on transparent, reflective, thin, and feature-less scene parts. FaDIV-Syn can perform both interpolation and extrapolation tasks at 540p in real-time and outperforms state-of-the-art extrapolation methods on the large-scale RealEstate10k dataset. We thoroughly evaluate ablations, such as removing the Soft-Masking network, training from fewer examples as well as generalization to higher resolutions and stronger depth discretization. Our implementation is available.