# Depth-Aware Video Frame Interpolation

arXiv: https://arxiv.org/abs/1904.00830 · HTML: https://mlsanity.com/paper/1904.00830/

## Key facts

- Cited by 234 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2019.

## Built on by (most cited first, 30 of 234)

- [MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation](https://mlsanity.com/paper/2205.09853/) (2022)
- [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](https://mlsanity.com/paper/2112.14683/) (2021)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [VRT: A Video Restoration Transformer](https://mlsanity.com/paper/2201.12288/) (2022)
- [AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation](https://mlsanity.com/paper/2304.09790/) (2023)
- [DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion](https://mlsanity.com/paper/2304.06025/) (2023)
- [FILM: Frame Interpolation for Large Motion](https://mlsanity.com/paper/2202.04901/) (2022)
- [TimeLens: Event-based Video Frame Interpolation](https://mlsanity.com/paper/2106.07286/) (2021)
- [XVFI: eXtreme Video Frame Interpolation](https://mlsanity.com/paper/2103.16206/) (2021)
- [Video Frame Interpolation Transformer](https://mlsanity.com/paper/2111.13817/) (2021)
- [Time Lens++: Event-based Frame Interpolation with Parametric Non-linear Flow and Multi-scale Fusion](https://mlsanity.com/paper/2203.17191/) (2022)
- [IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation](https://mlsanity.com/paper/2205.14620/) (2022)
- [Asymmetric Bilateral Motion Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2108.06815/) (2021)
- [VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution](https://mlsanity.com/paper/2206.04647/) (2022)
- [Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation](https://mlsanity.com/paper/2303.00440/) (2023)
- [LDMVFI: Video Frame Interpolation with Latent Diffusion Models](https://mlsanity.com/paper/2303.09508/) (2023)
- [Only Train Once: A One-Shot Neural Network Training And Pruning Framework](https://mlsanity.com/paper/2107.07467/) (2021)
- [ToonCrafter: Generative Cartoon Interpolation](https://mlsanity.com/paper/2405.17933/) (2024)
- [Real-World Single Image Super-Resolution: A Brief Review](https://mlsanity.com/paper/2103.02368/) (2021)
- [Deep Animation Video Interpolation in the Wild](https://mlsanity.com/paper/2104.02495/) (2021)
- [Unifying Motion Deblurring and Frame Interpolation with Events](https://mlsanity.com/paper/2203.12178/) (2022)
- [CDFI: Compression-Driven Network Design for Frame Interpolation](https://mlsanity.com/paper/2103.10559/) (2021)
- [Temporal Modulation Network for Controllable Space-Time Video Super-Resolution](https://mlsanity.com/paper/2104.10642/) (2021)
- [Stochastic Image-to-Video Synthesis using cINNs](https://mlsanity.com/paper/2105.04551/) (2021)
- [AdaPool: Exponential Adaptive Pooling for Information-Retaining Downsampling](https://mlsanity.com/paper/2111.00772/) (2021)
- [RSTT: Real-time Spatial Temporal Transformer for Space-Time Video Super-Resolution](https://mlsanity.com/paper/2203.14186/) (2022)
- [iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis](https://mlsanity.com/paper/2107.02790/) (2021)
- [Many-to-many Splatting for Efficient Video Frame Interpolation](https://mlsanity.com/paper/2204.03513/) (2022)
- [Flow supervision for Deformable NeRF](https://mlsanity.com/paper/2303.16333/) (2023)
- [MGMAE: Motion Guided Masking for Video Masked Autoencoding](https://mlsanity.com/paper/2308.10794/) (2023)

## Abstract (arXiv)

Video frame interpolation aims to synthesize nonexistent frames in-between the original frames. While significant advances have been made from the recent deep convolutional neural networks, the quality of interpolation is often reduced due to large object motion or occlusion. In this work, we propose a video frame interpolation method which explicitly detects the occlusion by exploring the depth information. Specifically, we develop a depth-aware flow projection layer to synthesize intermediate flows that preferably sample closer objects than farther ones. In addition, we learn hierarchical features to gather contextual information from neighboring pixels. The proposed model then warps the input frames, depth maps, and contextual features based on the optical flow and local interpolation kernels for synthesizing the output frame. Our model is compact, efficient, and fully differentiable. Quantitative and qualitative results demonstrate that the proposed model performs favorably against state-of-the-art frame interpolation methods on a wide variety of datasets.