# AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation

arXiv: https://arxiv.org/abs/2304.09790 · HTML: https://mlsanity.com/paper/2304.09790/

## Key facts

- Cited by 180 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 85 references that are in the map.
- First posted on arXiv 19 Apr 2023.

## Builds on (most cited first, 30 of 63)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching](https://mlsanity.com/paper/2109.07547/) (2021)
- [BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment](https://mlsanity.com/paper/2104.13371/) (2021)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [Video Frame Interpolation via Adaptive Separable Convolution](https://mlsanity.com/paper/1708.01692/) (2017)
- [Real-Time Intermediate Flow Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2011.06294/) (2020)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [DeepStereo: Learning to Predict New Views from the World's Imagery](https://mlsanity.com/paper/1506.06825/) (2015)
- [View Synthesis by Appearance Flow](https://mlsanity.com/paper/1605.03557/) (2016)
- [Video Frame Interpolation via Adaptive Convolution](https://mlsanity.com/paper/1703.07514/) (2017)

## Built on by (most cited first, 30 of 180)

- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Taming Rectified Flow for Inversion and Editing](https://mlsanity.com/paper/2411.04746/) (2024)
- [MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions](https://mlsanity.com/paper/2407.06358/) (2024)
- [SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency](https://mlsanity.com/paper/2407.17470/) (2024)
- [ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation](https://mlsanity.com/paper/2402.04324/) (2024)
- [VideoDPO: Omni-Preference Alignment for Video Diffusion Generation](https://mlsanity.com/paper/2412.14167/) (2024)
- [FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention](https://mlsanity.com/paper/2407.19918/) (2024)
- [T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback](https://mlsanity.com/paper/2405.18750/) (2024)
- [Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator](https://mlsanity.com/paper/2309.14494/) (2023)
- [Video Interpolation with Diffusion Models](https://mlsanity.com/paper/2404.01203/) (2024)
- [GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning](https://mlsanity.com/paper/2311.12631/) (2023)
- [Dense Optical Tracking: Connecting the Dots](https://mlsanity.com/paper/2312.00786/) (2023)
- [COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing](https://mlsanity.com/paper/2406.08850/) (2024)
- [Video-Bench: Human-Aligned Video Generation Benchmark](https://mlsanity.com/paper/2504.04907/) (2025)
- [Animate3D: Animating Any 3D Model with Multi-view Video Diffusion](https://mlsanity.com/paper/2407.11398/) (2024)
- [T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Design](https://mlsanity.com/paper/2410.05677/) (2024)
- [MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators](https://mlsanity.com/paper/2404.05014/) (2024)
- [MarDini: Masked Autoregressive Diffusion for Video Generation at Scale](https://mlsanity.com/paper/2410.20280/) (2024)
- [VMBench: A Benchmark for Perception-Aligned Video Motion Generation](https://mlsanity.com/paper/2503.10076/) (2025)
- [StyleMaster: Stylize Your Video with Artistic Generation and Translation](https://mlsanity.com/paper/2412.07744/) (2024)
- [Mora: Enabling Generalist Video Generation via A Multi-Agent Framework](https://mlsanity.com/paper/2403.13248/) (2024)
- [CorrMatch: Label Propagation via Correlation Matching for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2306.04300/) (2023)
- [Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation](https://mlsanity.com/paper/2412.06016/) (2024)
- [DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors](https://mlsanity.com/paper/2406.01476/) (2024)
- [EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation](https://mlsanity.com/paper/2411.08380/) (2024)
- [Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search](https://mlsanity.com/paper/2501.19252/) (2025)
- [Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations](https://mlsanity.com/paper/2507.00990/) (2025)
- [MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation](https://mlsanity.com/paper/2411.15262/) (2024)
- [VideoShield: Regulating Diffusion-based Video Generation Models via Watermarking](https://mlsanity.com/paper/2501.14195/) (2025)
- [PAI-Bench: A Comprehensive Benchmark For Physical AI](https://mlsanity.com/paper/2512.01989/) (2025)

## Abstract (arXiv)

We present All-Pairs Multi-Field Transforms (AMT), a new network architecture for video frame interpolation. It is based on two essential designs. First, we build bidirectional correlation volumes for all pairs of pixels, and use the predicted bilateral flows to retrieve correlations for updating both flows and the interpolated content feature. Second, we derive multiple groups of fine-grained flow fields from one pair of updated coarse flows for performing backward warping on the input frames separately. Combining these two designs enables us to generate promising task-oriented flows and reduce the difficulties in modeling large motions and handling occluded areas during frame interpolation. These qualities promote our model to achieve state-of-the-art performance on various benchmarks with high efficiency. Moreover, our convolution-based model competes favorably compared to Transformer-based models in terms of accuracy and efficiency. Our code is available at https://github.com/MCG-NKU/AMT.