# Video Frame Synthesis using Deep Voxel Flow

arXiv: https://arxiv.org/abs/1702.02463 · HTML: https://mlsanity.com/paper/1702.02463/

## Key facts

- Cited by 209 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2017.

## Built on by (most cited first, 30 of 209)

- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2412.07772/) (2024)
- [Diffusion Probabilistic Modeling for Video Generation](https://mlsanity.com/paper/2203.09481/) (2022)
- [VRT: A Video Restoration Transformer](https://mlsanity.com/paper/2201.12288/) (2022)
- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation](https://mlsanity.com/paper/2304.09790/) (2023)
- [FILM: Frame Interpolation for Large Motion](https://mlsanity.com/paper/2202.04901/) (2022)
- [TimeLens: Event-based Video Frame Interpolation](https://mlsanity.com/paper/2106.07286/) (2021)
- [XVFI: eXtreme Video Frame Interpolation](https://mlsanity.com/paper/2103.16206/) (2021)
- [IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation](https://mlsanity.com/paper/2205.14620/) (2022)
- [Asymmetric Bilateral Motion Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2108.06815/) (2021)
- [Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation](https://mlsanity.com/paper/2303.00440/) (2023)
- [LDMVFI: Video Frame Interpolation with Latent Diffusion Models](https://mlsanity.com/paper/2303.09508/) (2023)
- [LAPAR: Linearly-Assembled Pixel-Adaptive Regression Network for Single Image Super-Resolution and Beyond](https://mlsanity.com/paper/2105.10422/) (2021)
- [FitVid: Overfitting in Pixel-Level Video Prediction](https://mlsanity.com/paper/2106.13195/) (2021)
- [Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2206.12126/) (2022)
- [Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction](https://mlsanity.com/paper/2103.04174/) (2021)
- [Deep Animation Video Interpolation in the Wild](https://mlsanity.com/paper/2104.02495/) (2021)
- [High-resolution Face Swapping via Latent Semantics Disentanglement](https://mlsanity.com/paper/2203.15958/) (2022)
- [CDFI: Compression-Driven Network Design for Frame Interpolation](https://mlsanity.com/paper/2103.10559/) (2021)
- [MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions](https://mlsanity.com/paper/2103.02243/) (2021)
- [OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning](https://mlsanity.com/paper/2306.11249/) (2023)
- [A Dynamic Multi-Scale Voxel Flow Network for Video Prediction](https://mlsanity.com/paper/2303.09875/) (2023)
- [RSTT: Real-time Spatial Temporal Transformer for Space-Time Video Super-Resolution](https://mlsanity.com/paper/2203.14186/) (2022)
- [PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Prediction](https://mlsanity.com/paper/2305.11421/) (2023)
- [Many-to-many Splatting for Efficient Video Frame Interpolation](https://mlsanity.com/paper/2204.03513/) (2022)
- [A Unified Pyramid Recurrent Network for Video Frame Interpolation](https://mlsanity.com/paper/2211.03456/) (2022)
- [Understanding Object Dynamics for Interactive Image-to-Video Synthesis](https://mlsanity.com/paper/2106.11303/) (2021)
- [TimeReplayer: Unlocking the Potential of Event Cameras for Video Interpolation](https://mlsanity.com/paper/2203.13859/) (2022)
- [FloLPIPS: A Bespoke Video Quality Metric for Frame Interpoation](https://mlsanity.com/paper/2207.08119/) (2022)
- [ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation](https://mlsanity.com/paper/2111.15483/) (2021)

## Abstract (arXiv)

We address the problem of synthesizing new video frames in an existing video, either in-between existing frames (interpolation), or subsequent to them (extrapolation). This problem is challenging because video appearance and motion can be highly complex. Traditional optical-flow-based solutions often fail where flow estimation is challenging, while newer neural-network-based methods that hallucinate pixel values directly often produce blurry results. We combine the advantages of these two methods by training a deep network that learns to synthesize video frames by flowing pixel values from existing ones, which we call deep voxel flow. Our method requires no human supervision, and any video can be used as training data by dropping, and then learning to predict, existing frames. The technique is efficient, and can be applied at any video resolution. We demonstrate that our method produces results that both quantitatively and qualitatively improve upon the state-of-the-art.