mlsanityData to 7 Oct 2026

Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation

arXiv 2408.15239 ↗First posted 27 Aug 2024cs.CVcited by 42

On the map: Diffusion models → Video generation

Abstract (arXiv)

We present a method for generating video sequences with coherent motion between a pair of input key frames. We adapt a pretrained large-scale image-to-video diffusion model (originally trained to generate videos moving forward in time from a single input image) for key frame interpolation, i.e., to produce a video in between two input frames. We accomplish this adaptation through a lightweight fine-tuning technique that produces a version of the model that instead predicts videos moving backwards in time from a single input image. This model (along with the original forward-moving model) is subsequently used in a dual-directional diffusion sampling process that combines the overlapping model estimates starting from each of the two keyframes. Our experiments show that our method outperforms both existing diffusion-based methods and traditional frame interpolation techniques.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 34 of its 39 references that are in the map. First posted on arXiv 27 Aug 2024.

Builds on

34 of 39 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Video Interpolation with Diffusion Models1 Apr
  2. 2024Lumiere: A Space-Time Diffusion Model for Video Generation23 Jan
  3. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models28 Nov
  4. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors18 Oct
  5. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  6. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  8. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
  9. 2022Video Diffusion Models7 Apr

Other references · 25

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Denoising Diffusion Implicit Modelscited by 9,655
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  8. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  9. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  10. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  11. 2023Universal Guidance for Diffusion Modelscited by 426
  12. 2017Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolationcited by 321
13 more
  1. 2020Softmax Splatting for Video Frame Interpolationcited by 276
  2. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 210
  3. 2022FILM: Frame Interpolation for Large Motioncited by 149
  4. 2019Quadratic video interpolationcited by 143
  5. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  6. 2019AdaCoF: Adaptive Collaboration of Flows for Video Frame Interpolationcited by 133
  7. 2021XVFI: eXtreme Video Frame Interpolationcited by 123
  8. 2020BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolationcited by 117
  9. 2023SyncDiffusion: Coherent Montage via Synchronized Joint Diffusionscited by 108
  10. 2021Asymmetric Bilateral Motion Estimation for Video Frame Interpolationcited by 101
  11. 2023DiffCollage: Parallel Generation of Large Content with Diffusion Modelscited by 100
  12. 2020Enhanced Quadratic Video Interpolationcited by 48
  13. 2019Unsupervised Video Interpolation Using Cycle Consistencycited by 44

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 18; 2026: 16.

42 papers.

Sort:

By area:

  1. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 100
  2. 2025Let Them Talk: Audio-Driven Multi-Person Conversational Video Generationcited by 64
  3. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49
  4. 2024Framer: Interactive Frame Interpolationcited by 24
  5. 2024Generative Inbetweening through Frame-wise Conditions-Driven Video Generationcited by 23
  6. 2024ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Samplercited by 19
  7. 2024Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep Approachcited by 18
  8. 2025Kontinuous Kontext: Continuous Strength Control for Instruction-based Image Editingcited by 15
  9. 2025Motion-Aware Generative Frame Interpolationcited by 13
  10. 2026Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoningcited by 13
  11. 2025Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Modelscited by 11
  12. 2026PISCO: Precise Video Instance Insertion with Sparse Controlcited by 9
30 more
  1. 2025AceVFI: A Comprehensive Survey of Advances in Video Frame Interpolationcited by 8
  2. 2024KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photoscited by 7
  3. 2025Versatile Transition Generation with Image-to-Video Diffusioncited by 7
  4. 2025EGVD: Event-Guided Video Diffusion Model for Physically Realistic Large-Motion Frame Interpolationcited by 6
  5. 2026Motion Prior Distillation in Time Reversal Sampling for Generative Inbetweeningcited by 5
  6. 2025Controllable Human-centric Keyframe Interpolation with Generative Priorcited by 4
  7. 2025Pusa V1.0: Unlocking Temporal Control in Pretrained Video Diffusion Models via Vectorized Timestep Adaptationcited by 4
  8. 2026UniTemp: Unlocking Video Generation in Any Temporal Order via Bidirectional Distillationcited by 4
  9. 2025EF-VI: Enhancing End-Frame Injection for Video Inbetweeningcited by 3
  10. 2025FakeParts: a New Family of AI-Generated DeepFakescited by 3
  11. 2025SeeU: Seeing the Unseen World via 4D Dynamics-aware Generationcited by 3
  12. 2026SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learningcited by 2
  13. 2025DiffuseSlide: Training-Free High Frame Rate Video Generation Diffusioncited by 1
  14. 2025Semantic Frame Interpolationcited by 1
  15. 2025Arbitrary Generative Video Interpolationcited by 1
  16. 2025Time-Correlated Video Bridge Matchingcited by 1
  17. 2026Anchoring and Rescaling Attention for Semantically Coherent Inbetweeningcited by 1
  18. 2026Can Video Diffusion Models Predict Past Frames? Bidirectional Cycle Consistency for Reversible Interpolationcited by 1
  19. 2026Seeing Fast and Slow: Learning the Flow of Time in Videoscited by 1
  20. 2024Adapting Image-to-Video Diffusion Models for Large-Motion Frame Interpolationcited by 0
  21. 2025How Animals Dance (When You're Not Looking)cited by 0
  22. 2025A Study of the Design Space of Motion and Disocclusion Control in Video Generationcited by 0
  23. 2026FREE-Edit: Using Editing-aware Injection in Rectified Flow Models for Zero-shot Image-Driven Video Editingcited by 0
  24. 2026FC-VFI: Faithful and Consistent Video Frame Interpolation for High-FPS Slow Motion Video Generationcited by 0
  25. 2026Ego-InBetween: Generating Object State Transitions in Ego-Centric Videoscited by 0
  26. 2026DiffST: Spatiotemporal-Aware Diffusion for Real-World Space-Time Video Super-Resolutioncited by 0
  27. 2026SketchKeyAnime: Reference-anchored Sparse Key-Sketch Animation Synthesiscited by 0
  28. 2026Adaptive Identity Anchoring: Closed-Loop Keyframe Placement for Synthetic Paired Supervision in Video Face Swappingcited by 0
  29. 2026Bridging Event Streams and DiT: Event-Guided Video Frame Interpolationcited by 0
  30. 2026SNM-VFI: Symmetric Nonlinear Motion-Guided Generative Video Frame Interpolationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Less is More: Improving Motion Diffusion Models with Sparse Keyframes
  2. 2025EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation
  3. 2026Efficient Camera-Controlled Video Generation of Static Scenes via Sparse Diffusion and 3D Rendering
  4. 2026Efficient Video Diffusion Models: Advancements and Challenges
  5. 2026SPEED: One-Step Pixel Diffusion for High-quality Video Frame Interpolation
  6. 2024Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition
  7. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers
  8. 2026Towards Holistic Modeling for Video Frame Interpolation with Auto-regressive Diffusion Transformers