# Enhanced Correlation Matching based Video Frame Interpolation

arXiv: https://arxiv.org/abs/2111.08869 · HTML: https://mlsanity.com/paper/2111.08869/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 17 Nov 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [Video Frame Interpolation via Adaptive Separable Convolution](https://mlsanity.com/paper/1708.01692/) (2017)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [DeepStereo: Learning to Predict New Views from the World's Imagery](https://mlsanity.com/paper/1506.06825/) (2015)
- [Quadratic video interpolation](https://mlsanity.com/paper/1911.00627/) (2019)
- [AdaCoF: Adaptive Collaboration of Flows for Video Frame Interpolation](https://mlsanity.com/paper/1907.10244/) (2019)
- [XVFI: eXtreme Video Frame Interpolation](https://mlsanity.com/paper/2103.16206/) (2021)
- [MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement](https://mlsanity.com/paper/1810.08768/) (2018)
- [Accurate Optical Flow via Direct Cost Volume Processing](https://mlsanity.com/paper/1704.07325/) (2017)
- [Learning Image Matching by Simply Watching Video](https://mlsanity.com/paper/1603.06041/) (2016)

## Built on by (most cited first, 7 of 7)

- [Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation](https://mlsanity.com/paper/2303.00440/) (2023)
- [A Unified Pyramid Recurrent Network for Video Frame Interpolation](https://mlsanity.com/paper/2211.03456/) (2022)
- [Enhanced Bi-directional Motion Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2206.08572/) (2022)
- [Data-Efficient Unsupervised Interpolation Without Any Intermediate Frame for 4D Medical Images](https://mlsanity.com/paper/2404.01464/) (2024)
- [Frame Interpolation for Dynamic Scenes with Implicit Flow Encoding](https://mlsanity.com/paper/2209.13284/) (2022)
- [Motion-Aware Video Frame Interpolation](https://mlsanity.com/paper/2402.02892/) (2024)
- [Restoration-Oriented Video Frame Interpolation with Region-Distinguishable Priors from SAM](https://mlsanity.com/paper/2312.15868/) (2023)

## Abstract (arXiv)

We propose a novel DNN based framework called the Enhanced Correlation Matching based Video Frame Interpolation Network to support high resolution like 4K, which has a large scale of motion and occlusion. Considering the extensibility of the network model according to resolution, the proposed scheme employs the recurrent pyramid architecture that shares the parameters among each pyramid layer for optical flow estimation. In the proposed flow estimation, the optical flows are recursively refined by tracing the location with maximum correlation. The forward warping based correlation matching enables to improve the accuracy of flow update by excluding incorrectly warped features around the occlusion area. Based on the final bi-directional flows, the intermediate frame at arbitrary temporal position is synthesized using the warping and blending network and it is further improved by refinement network. Experiment results demonstrate that the proposed scheme outperforms the previous works at 4K video data and low-resolution benchmark datasets as well in terms of objective and subjective quality with the smallest number of model parameters.