# Unsupervised Video Interpolation by Learning Multilayered 2.5D Motion Fields

arXiv: https://arxiv.org/abs/2204.09900 · HTML: https://mlsanity.com/paper/2204.09900/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 21 Apr 2022.

## Builds on (most cited first, 29 of 29)

- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [Learned Initializations for Optimizing Coordinate-Based Neural Representations](https://mlsanity.com/paper/2012.02189/) (2020)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [Video Frame Interpolation via Adaptive Separable Convolution](https://mlsanity.com/paper/1708.01692/) (2017)
- [Stereo Magnification: Learning View Synthesis using Multiplane Images](https://mlsanity.com/paper/1805.09817/) (2018)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [Video Frame Interpolation via Adaptive Convolution](https://mlsanity.com/paper/1703.07514/) (2017)
- [Context-aware Synthesis for Video Frame Interpolation](https://mlsanity.com/paper/1803.10967/) (2018)
- [Quadratic video interpolation](https://mlsanity.com/paper/1911.00627/) (2019)
- [XVFI: eXtreme Video Frame Interpolation](https://mlsanity.com/paper/2103.16206/) (2021)
- [BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolation](https://mlsanity.com/paper/2007.12622/) (2020)
- [MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement](https://mlsanity.com/paper/1810.08768/) (2018)
- [Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers](https://mlsanity.com/paper/2103.12340/) (2021)
- [Asymmetric Bilateral Motion Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2108.06815/) (2021)
- [Betrayed by Motion: Camouflaged Object Discovery via Motion Segmentation](https://mlsanity.com/paper/2011.11630/) (2020)
- [Blurry Video Frame Interpolation](https://mlsanity.com/paper/2002.12259/) (2020)
- [Enhanced Quadratic Video Interpolation](https://mlsanity.com/paper/2009.04642/) (2020)
- [Unsupervised Video Interpolation Using Cycle Consistency](https://mlsanity.com/paper/1906.05928/) (2019)
- [Learning Image Matching by Simply Watching Video](https://mlsanity.com/paper/1603.06041/) (2016)
- [The Visual Centrifuge: Model-Free Layered Video Representations](https://mlsanity.com/paper/1812.01461/) (2018)
- [Fast Soft Color Segmentation](https://mlsanity.com/paper/2004.08096/) (2020)
- [One Ring to Rule Them All: a simple solution to multi-view 3D-Reconstruction of shapes with unknown BRDF via a small Recurrent ResNet](https://mlsanity.com/paper/2104.05014/) (2021)

## Built on by (most cited first, 1 of 1)

- [BVI-VFI: A Video Quality Database for Video Frame Interpolation](https://mlsanity.com/paper/2210.00823/) (2022)

## Abstract (arXiv)

The problem of video frame interpolation is to increase the temporal resolution of a low frame-rate video, by interpolating novel frames between existing temporally sparse frames. This paper presents a self-supervised approach to video frame interpolation that requires only a single video. We pose the video as a set of layers. Each layer is parameterized by two implicit neural networks -- one for learning a static frame and the other for a time-varying motion field corresponding to video dynamics. Together they represent an occlusion-free subset of the scene with a pseudo-depth channel. To model inter-layer occlusions, all layers are lifted to the 2.5D space so that the frontal layer occludes distant layers. This is done by assigning each layer a depth channel, which we call `pseudo-depth', whose partial order defines the occlusion between layers. The pseudo-depths are converted to visibility values through a fully differentiable SoftMin function so that closer layers are more visible than layers in a distance. On the other hand, we parameterize the video motions by solving an ordinary differentiable equation (ODE) defined on a time-varying neural velocity field that guarantees valid motions. This implicit neural representation learns the video as a space-time continuum, allowing frame interpolation at any temporal resolution. We demonstrate the effectiveness of our method on real-world datasets, where our method achieves comparable performance to state-of-the-arts that require ground truth labels for training.