# Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation

arXiv: https://arxiv.org/abs/2408.15239 · HTML: https://mlsanity.com/paper/2408.15239/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 34 of its 39 references that are in the map.
- First posted on arXiv 27 Aug 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 34)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [Universal Guidance for Diffusion Models](https://mlsanity.com/paper/2302.07121/) (2023)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation](https://mlsanity.com/paper/2205.09853/) (2022)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion](https://mlsanity.com/paper/2307.01097/) (2023)
- [SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models](https://mlsanity.com/paper/2311.16933/) (2023)
- [FILM: Frame Interpolation for Large Motion](https://mlsanity.com/paper/2202.04901/) (2022)
- [Quadratic video interpolation](https://mlsanity.com/paper/1911.00627/) (2019)
- [Make Pixels Dance: High-Dynamic Video Generation](https://mlsanity.com/paper/2311.10982/) (2023)
- [AdaCoF: Adaptive Collaboration of Flows for Video Frame Interpolation](https://mlsanity.com/paper/1907.10244/) (2019)
- [XVFI: eXtreme Video Frame Interpolation](https://mlsanity.com/paper/2103.16206/) (2021)
- [BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolation](https://mlsanity.com/paper/2007.12622/) (2020)
- [SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions](https://mlsanity.com/paper/2306.05178/) (2023)
- [DiffCollage: Parallel Generation of Large Content with Diffusion Models](https://mlsanity.com/paper/2303.17076/) (2023)

## Built on by (most cited first, 30 of 44)

- [Adaptive Caching for Faster Video Generation with Diffusion Transformers](https://mlsanity.com/paper/2411.02397/) (2024)
- [Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation](https://mlsanity.com/paper/2505.22647/) (2025)
- [MarDini: Masked Autoregressive Diffusion for Video Generation at Scale](https://mlsanity.com/paper/2410.20280/) (2024)
- [Framer: Interactive Frame Interpolation](https://mlsanity.com/paper/2410.18978/) (2024)
- [Generative Inbetweening through Frame-wise Conditions-Driven Video Generation](https://mlsanity.com/paper/2412.11755/) (2024)
- [ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler](https://mlsanity.com/paper/2410.05651/) (2024)
- [Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep Approach](https://mlsanity.com/paper/2410.03160/) (2024)
- [Kontinuous Kontext: Continuous Strength Control for Instruction-based Image Editing](https://mlsanity.com/paper/2510.08532/) (2025)
- [Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning](https://mlsanity.com/paper/2603.19607/) (2026)
- [Motion-Aware Generative Frame Interpolation](https://mlsanity.com/paper/2501.03699/) (2025)
- [Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Models](https://mlsanity.com/paper/2506.07177/) (2025)
- [PISCO: Precise Video Instance Insertion with Sparse Control](https://mlsanity.com/paper/2602.08277/) (2026)
- [AceVFI: A Comprehensive Survey of Advances in Video Frame Interpolation](https://mlsanity.com/paper/2506.01061/) (2025)
- [KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photos](https://mlsanity.com/paper/2411.13549/) (2024)
- [Versatile Transition Generation with Image-to-Video Diffusion](https://mlsanity.com/paper/2508.01698/) (2025)
- [EGVD: Event-Guided Video Diffusion Model for Physically Realistic Large-Motion Frame Interpolation](https://mlsanity.com/paper/2503.20268/) (2025)
- [Motion Prior Distillation in Time Reversal Sampling for Generative Inbetweening](https://mlsanity.com/paper/2602.12679/) (2026)
- [SeeU: Seeing the Unseen World via 4D Dynamics-aware Generation](https://mlsanity.com/paper/2512.03350/) (2025)
- [EF-VI: Enhancing End-Frame Injection for Video Inbetweening](https://mlsanity.com/paper/2505.21205/) (2025)
- [Controllable Human-centric Keyframe Interpolation with Generative Prior](https://mlsanity.com/paper/2506.03119/) (2025)
- [Pusa V1.0: Unlocking Temporal Control in Pretrained Video Diffusion Models via Vectorized Timestep Adaptation](https://mlsanity.com/paper/2507.16116/) (2025)
- [UniTemp: Unlocking Video Generation in Any Temporal Order via Bidirectional Distillation](https://mlsanity.com/paper/2606.18702/) (2026)
- [FakeParts: a New Family of AI-Generated DeepFakes](https://mlsanity.com/paper/2508.21052/) (2025)
- [Time-Correlated Video Bridge Matching](https://mlsanity.com/paper/2510.12453/) (2025)
- [SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learning](https://mlsanity.com/paper/2603.10446/) (2026)
- [Anchoring and Rescaling Attention for Semantically Coherent Inbetweening](https://mlsanity.com/paper/2603.17651/) (2026)
- [Audio-Visual Intelligence in Large Foundation Models](https://mlsanity.com/paper/2605.04045/) (2026)
- [Adapting Image-to-Video Diffusion Models for Large-Motion Frame Interpolation](https://mlsanity.com/paper/2412.17042/) (2024)
- [DiffuseSlide: Training-Free High Frame Rate Video Generation Diffusion](https://mlsanity.com/paper/2506.01454/) (2025)
- [Semantic Frame Interpolation](https://mlsanity.com/paper/2507.05173/) (2025)

## Abstract (arXiv)

We present a method for generating video sequences with coherent motion between a pair of input key frames. We adapt a pretrained large-scale image-to-video diffusion model (originally trained to generate videos moving forward in time from a single input image) for key frame interpolation, i.e., to produce a video in between two input frames. We accomplish this adaptation through a lightweight fine-tuning technique that produces a version of the model that instead predicts videos moving backwards in time from a single input image. This model (along with the original forward-moving model) is subsequently used in a dual-directional diffusion sampling process that combines the overlapping model estimates starting from each of the two keyframes. Our experiments show that our method outperforms both existing diffusion-based methods and traditional frame interpolation techniques.