# Revealing Disocclusions in Temporal View Synthesis through Infilling Vector Prediction

arXiv: https://arxiv.org/abs/2110.08805 · HTML: https://mlsanity.com/paper/2110.08805/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 17 Oct 2021.

## Builds on (most cited first, 21 of 21)

- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [DeepView: View Synthesis with Learned Gradient Descent](https://mlsanity.com/paper/1906.07316/) (2019)
- [Stereo Magnification: Learning View Synthesis using Multiplane Images](https://mlsanity.com/paper/1805.09817/) (2018)
- [3D Photography using Context-aware Layered Depth Inpainting](https://mlsanity.com/paper/2004.04727/) (2020)
- [Pushing the Boundaries of View Extrapolation with Multiplane Images](https://mlsanity.com/paper/1905.00413/) (2019)
- [View Synthesis by Appearance Flow](https://mlsanity.com/paper/1605.03557/) (2016)
- [SelFlow: Self-Supervised Learning of Optical Flow](https://mlsanity.com/paper/1904.09117/) (2019)
- [Deep Flow-Guided Video Inpainting](https://mlsanity.com/paper/1905.02884/) (2019)
- [A Review on Deep Learning Techniques for Video Prediction](https://mlsanity.com/paper/2004.05214/) (2020)
- [Deep Video Inpainting](https://mlsanity.com/paper/1905.01639/) (2019)
- [Recurrent Feature Reasoning for Image Inpainting](https://mlsanity.com/paper/2008.03737/) (2020)
- [Flow-edge Guided Video Completion](https://mlsanity.com/paper/2009.01835/) (2020)
- [Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness](https://mlsanity.com/paper/1608.05842/) (2016)
- [Occlusion Aware Unsupervised Learning of Optical Flow](https://mlsanity.com/paper/1711.05890/) (2017)
- [Extreme View Synthesis](https://mlsanity.com/paper/1812.04777/) (2018)
- [High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling](https://mlsanity.com/paper/2005.11742/) (2020)
- [Transformation-Grounded Image Generation Network for Novel 3D View Synthesis](https://mlsanity.com/paper/1703.02921/) (2017)
- [DVI: Depth Guided Video Inpainting for Autonomous Driving](https://mlsanity.com/paper/2007.08854/) (2020)

## Built on by (most cited first, 5 of 5)

- [SimpleNeRF: Regularizing Sparse Input Neural Radiance Fields with Simpler Solutions](https://mlsanity.com/paper/2309.03955/) (2023)
- [MVPGS: Excavating Multi-view Priors for Gaussian Splatting from Sparse Input Views](https://mlsanity.com/paper/2409.14316/) (2024)
- [Low Light Video Enhancement by Learning on Static Videos with Cross-Frame Attention](https://mlsanity.com/paper/2210.04290/) (2022)
- [Temporal View Synthesis of Dynamic Scenes through 3D Object Motion Estimation with Multi-Plane Images](https://mlsanity.com/paper/2208.09463/) (2022)
- [Simple-RF: Regularizing Sparse Input Radiance Fields with Simpler Solutions](https://mlsanity.com/paper/2404.19015/) (2024)

## Abstract (arXiv)

We consider the problem of temporal view synthesis, where the goal is to predict a future video frame from the past frames using knowledge of the depth and relative camera motion. In contrast to revealing the disoccluded regions through intensity based infilling, we study the idea of an infilling vector to infill by pointing to a non-disoccluded region in the synthesized view. To exploit the structure of disocclusions created by camera motion during their infilling, we rely on two important cues, temporal correlation of infilling directions and depth. We design a learning framework to predict the infilling vector by computing a temporal prior that reflects past infilling directions and a normalized depth map as input to the network. We conduct extensive experiments on a large scale dataset we build for evaluating temporal view synthesis in addition to the SceneNet RGB-D dataset. Our experiments demonstrate that our infilling vector prediction approach achieves superior quantitative and qualitative infilling performance compared to other approaches in literature.