# Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation

arXiv: https://arxiv.org/abs/2303.00440 · HTML: https://mlsanity.com/paper/2303.00440/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 62 references that are in the map.
- First posted on arXiv 1 Mar 2023.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases](https://mlsanity.com/paper/2103.10697/) (2021)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [Incorporating Convolution Designs into Visual Transformers](https://mlsanity.com/paper/2103.11816/) (2021)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [LocalViT: Analyzing Locality in Vision Transformers](https://mlsanity.com/paper/2104.05707/) (2021)
- [Video Frame Interpolation via Adaptive Separable Convolution](https://mlsanity.com/paper/1708.01692/) (2017)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [DeepStereo: Learning to Predict New Views from the World's Imagery](https://mlsanity.com/paper/1506.06825/) (2015)
- [Video Frame Interpolation via Adaptive Convolution](https://mlsanity.com/paper/1703.07514/) (2017)
- [Context-aware Synthesis for Video Frame Interpolation](https://mlsanity.com/paper/1803.10967/) (2018)
- [FILM: Frame Interpolation for Large Motion](https://mlsanity.com/paper/2202.04901/) (2022)
- [Video Compression through Image Interpolation](https://mlsanity.com/paper/1804.06919/) (2018)
- [AdaCoF: Adaptive Collaboration of Flows for Video Frame Interpolation](https://mlsanity.com/paper/1907.10244/) (2019)
- [XVFI: eXtreme Video Frame Interpolation](https://mlsanity.com/paper/2103.16206/) (2021)
- [Video Frame Interpolation Transformer](https://mlsanity.com/paper/2111.13817/) (2021)
- [BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolation](https://mlsanity.com/paper/2007.12622/) (2020)

## Built on by (most cited first, 30 of 98)

- [AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation](https://mlsanity.com/paper/2304.09790/) (2023)
- [Allegro: Open the Black Box of Commercial-Level Video Generation Model](https://mlsanity.com/paper/2410.15458/) (2024)
- [Video Interpolation with Diffusion Models](https://mlsanity.com/paper/2404.01203/) (2024)
- [ToonCrafter: Generative Cartoon Interpolation](https://mlsanity.com/paper/2405.17933/) (2024)
- [A Dynamic Multi-Scale Voxel Flow Network for Video Prediction](https://mlsanity.com/paper/2303.09875/) (2023)
- [MagicVideo-V2: Multi-Stage High-Aesthetic Video Generation](https://mlsanity.com/paper/2401.04468/) (2024)
- [MGMAE: Motion Guided Masking for Video Masked Autoencoding](https://mlsanity.com/paper/2308.10794/) (2023)
- [Sparse Global Matching for Video Frame Interpolation with Large Motion](https://mlsanity.com/paper/2404.06913/) (2024)
- [AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation](https://mlsanity.com/paper/2407.04603/) (2024)
- [DS-NeRV: Implicit Neural Video Representation with Decomposed Static and Dynamic Codes](https://mlsanity.com/paper/2403.15679/) (2024)
- [Generative Inbetweening through Frame-wise Conditions-Driven Video Generation](https://mlsanity.com/paper/2412.11755/) (2024)
- [Perception-Oriented Video Frame Interpolation via Asymmetric Blending](https://mlsanity.com/paper/2404.06692/) (2024)
- [Repurposing Pre-trained Video Diffusion Models for Event-based Video Interpolation](https://mlsanity.com/paper/2412.07761/) (2024)
- [VFIMamba: Video Frame Interpolation with State Space Models](https://mlsanity.com/paper/2407.02315/) (2024)
- [Dual DETRs for Multi-Label Temporal Action Detection](https://mlsanity.com/paper/2404.00653/) (2024)
- [EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation](https://mlsanity.com/paper/2503.15831/) (2025)
- [PMQ-VE: Progressive Multi-Frame Quantization for Video Enhancement](https://mlsanity.com/paper/2505.12266/) (2025)
- [Data-Efficient Unsupervised Interpolation Without Any Intermediate Frame for 4D Medical Images](https://mlsanity.com/paper/2404.01464/) (2024)
- [DreamMover: Leveraging the Prior of Diffusion Models for Image Interpolation with Large Motion](https://mlsanity.com/paper/2409.09605/) (2024)
- [ZeroI2V: Zero-Cost Adaptation of Pre-trained Transformers from Image to Video](https://mlsanity.com/paper/2310.01324/) (2023)
- [BiM-VFI: Bidirectional Motion Field-Guided Frame Interpolation for Video with Non-uniform Motions](https://mlsanity.com/paper/2412.11365/) (2024)
- [Motion-Aware Generative Frame Interpolation](https://mlsanity.com/paper/2501.03699/) (2025)
- [Long Video Diffusion Generation with Segmented Cross-Attention and Content-Rich Video Data Curation](https://mlsanity.com/paper/2412.01316/) (2024)
- [OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence](https://mlsanity.com/paper/2602.08683/) (2026)
- [SportsSloMo: A New Benchmark and Baselines for Human-centric Video Frame Interpolation](https://mlsanity.com/paper/2308.16876/) (2023)
- [OCAI: Improving Optical Flow Estimation by Occlusion and Consistency Aware Interpolation](https://mlsanity.com/paper/2403.18092/) (2024)
- [LAVIB: A Large-scale Video Interpolation Benchmark](https://mlsanity.com/paper/2406.09754/) (2024)
- [Disentangled Motion Modeling for Video Frame Interpolation](https://mlsanity.com/paper/2406.17256/) (2024)
- [Video Frame Interpolation with Stereo Event and Intensity Camera](https://mlsanity.com/paper/2307.08228/) (2023)
- [BF-STVSR: B-Splines and Fourier-Best Friends for High Fidelity Spatial-Temporal Video Super-Resolution](https://mlsanity.com/paper/2501.11043/) (2025)

## Abstract (arXiv)

Effectively extracting inter-frame motion and appearance information is important for video frame interpolation (VFI). Previous works either extract both types of information in a mixed way or elaborate separate modules for each type of information, which lead to representation ambiguity and low efficiency. In this paper, we propose a novel module to explicitly extract motion and appearance information via a unifying operation. Specifically, we rethink the information process in inter-frame attention and reuse its attention map for both appearance feature enhancement and motion information extraction. Furthermore, for efficient VFI, our proposed module could be seamlessly integrated into a hybrid CNN and Transformer architecture. This hybrid pipeline can alleviate the computational complexity of inter-frame attention as well as preserve detailed low-level structure information. Experimental results demonstrate that, for both fixed- and arbitrary-timestep interpolation, our method achieves state-of-the-art performance on various datasets. Meanwhile, our approach enjoys a lighter computation overhead over models with close performance. The source code and models are available at https://github.com/MCG-NJU/EMA-VFI.