mlsanityData to 7 Oct 2026

Video Interpolation with Diffusion Models

arXiv 2404.01203 ↗First posted 1 Apr 2024cs.CVcited by 67

On the map: Diffusion models → Video generation

Abstract (arXiv)

We present VIDIM, a generative model for video interpolation, which creates short videos given a start and end frame. In order to achieve high fidelity and generate motions unseen in the input data, VIDIM uses cascaded diffusion models to first generate the target video at low resolution, and then generate the high-resolution video conditioned on the low-resolution generated video. We compare VIDIM to previous state-of-the-art methods on video interpolation, and demonstrate how such works fail in most settings where the underlying motion is complex, nonlinear, or ambiguous while VIDIM can easily handle such cases. We additionally demonstrate how classifier-free guidance on the start and end frame and conditioning the super-resolution model on the original high-resolution frames without additional parameters unlocks high-fidelity results. VIDIM is fast to sample from as it jointly denoises all the frames to be generated, requires less than a billion parameters per diffusion model to produce compelling results, and still enjoys scalability and improved quality at larger parameter counts.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 50 of its 63 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

50 of 63 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  2. 2023LDMVFI: Video Frame Interpolation with Latent Diffusion Models16 Mar
  3. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
  4. 2022Video Diffusion Models7 Apr

Other references · 46

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2022Generative Adversarial Networkscited by 5,603
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
34 more
  1. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  2. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  4. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  5. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  6. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  7. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  9. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  10. 2021Variational Diffusion Modelscited by 1,145
  11. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  12. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  13. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  14. 2017Video Enhancement with Task-Oriented Flowcited by 612
  15. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  16. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 413
  17. 2020PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Modelscited by 358
  18. 2017Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolationcited by 321
  19. 2022Novel View Synthesis with Diffusion Modelscited by 299
  20. 2020Softmax Splatting for Video Frame Interpolationcited by 276
  21. 2020Real-Time Intermediate Flow Estimation for Video Frame Interpolationcited by 211
  22. 2023AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolationcited by 177
  23. 2017FSRNet: End-to-End Learning Face Super-Resolution with Facial Priorscited by 170
  24. 2022FILM: Frame Interpolation for Large Motioncited by 149
  25. 2019Quadratic video interpolationcited by 143
  26. 2023TryOnDiffusion: A Tale of Two UNetscited by 141
  27. 2021XVFI: eXtreme Video Frame Interpolationcited by 123
  28. 2022IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolationcited by 107
  29. 2021Asymmetric Bilateral Motion Estimation for Video Frame Interpolationcited by 101
  30. 2023Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolationcited by 97
  31. 2020Enhanced Quadratic Video Interpolationcited by 48
  32. 2023Denoising Diffusion Probabilistic Models for Robust Image Super-Resolution in the Wildcited by 46
  33. 2017Pixel Recursive Super Resolutioncited by 44
  34. 2021ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolationcited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 29; 2026: 17.

67 papers.

Sort:

By area:

  1. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  2. 2024ToonCrafter: Generative Cartoon Interpolationcited by 68
  3. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49
  4. 2024GenXD: Generating Any 3D and 4D Scenescited by 49
  5. 2024Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolationcited by 42
  6. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 40
  7. 2024TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolationcited by 32
  8. 2024Framer: Interactive Frame Interpolationcited by 24
  9. 2024Generative Inbetweening through Frame-wise Conditions-Driven Video Generationcited by 23
  10. 2024AniDoc: Animation Creation Made Easiercited by 22
  11. 2024GenLit: Reformulating Single-Image Relighting as Video Generationcited by 20
  12. 2024ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Samplercited by 19
38 more
  1. 2024FrameBridge: Improving Image-to-Video Generation with Bridge Modelscited by 19
  2. 2025EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolationcited by 19
  3. 2024UniPaint: Unified Space-time Video Inpainting via Mixture-of-Expertscited by 13
  4. 2025Motion-Aware Generative Frame Interpolationcited by 13
  5. 2025Frame In-N-Out: Unbounded Controllable Image-to-Video Generationcited by 10
  6. 2024KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photoscited by 7
  7. 2025StreamSplat: Towards Online Dynamic 3D Reconstruction from Uncalibrated Video Streamscited by 7
  8. 2025TLB-VFI: Temporal-Aware Latent Brownian Bridge Diffusion for Video Frame Interpolationcited by 7
  9. 2025Versatile Transition Generation with Image-to-Video Diffusioncited by 7
  10. 2025EGVD: Event-Guided Video Diffusion Model for Physically Realistic Large-Motion Frame Interpolationcited by 6
  11. 2025Video Motion Graphscited by 5
  12. 2025FB-Diff: Fourier Basis-guided Diffusion for Temporal Interpolation of 4D Medical Imagingcited by 5
  13. 2026Motion Prior Distillation in Time Reversal Sampling for Generative Inbetweeningcited by 5
  14. 2024MAVIN: Multi-Action Video Generation with Diffusion Models via Transition Video Infillingcited by 4
  15. 2025HOFAR: High-Order Augmentation of Flow Autoregressive Transformerscited by 4
  16. 2026Towards Holistic Modeling for Video Frame Interpolation with Auto-regressive Diffusion Transformerscited by 4
  17. 2025EventDiff: A Unified and Efficient Diffusion Model Framework for Event-based Video Frame Interpolationcited by 3
  18. 2025Temporal Differential Fields for 4D Motion Modeling via Image-to-Video Synthesiscited by 3
  19. 2025FakeParts: a New Family of AI-Generated DeepFakescited by 3
  20. 2024Artificial Intelligence for Biomedical Video Generationcited by 2
  21. 2025Time-adaptive Video Frame Interpolation based on Residual Diffusioncited by 2
  22. 2025SAGE: Structure-Aware Generative Video Transitions between Diverse Clipscited by 2
  23. 2026SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learningcited by 2
  24. 2024DINTR: Tracking via Diffusion-based Interpolationcited by 1
  25. 2024AerialGo: Walking-through City View Generation from Aerial Perspectivescited by 1
  26. 2025DiffuseSlide: Training-Free High Frame Rate Video Generation Diffusioncited by 1
  27. 2025LumosFlow: Motion-Guided Long Video Generationcited by 1
  28. 2025Semantic Frame Interpolationcited by 1
  29. 2025Arbitrary Generative Video Interpolationcited by 1
  30. 2025Time-Correlated Video Bridge Matchingcited by 1
  31. 2026Taming Real-World Space-Time Video Super-Resolution with One-Step Diffusioncited by 1
  32. 2026Edit2Interp: Adapting Image Foundation Models from Spatial Editing to Video Frame Interpolation with Few-Shot Learningcited by 1
  33. 2026Can Video Diffusion Models Predict Past Frames? Bidirectional Cycle Consistency for Reversible Interpolationcited by 1
  34. 2026Seeing Fast and Slow: Learning the Flow of Time in Videoscited by 1
  35. 2026SPEED: One-Step Pixel Diffusion for High-quality Video Frame Interpolationcited by 1
  36. 2024From Diffusion to Resolution: Leveraging 2D Diffusion Models for 3D Super-Resolution Taskcited by 0
  37. 2024Factorized Video Autoencoders for Efficient Generative Modellingcited by 0
  38. 2024Adapting Image-to-Video Diffusion Models for Large-Motion Frame Interpolationcited by 0

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022VIDM: Video Implicit Diffusion Models
  2. 2023Video Probabilistic Diffusion Models in Projected Latent Space
  3. 2024ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video Generation
  4. 2026Efficient Video Diffusion Models: Advancements and Challenges
  5. 2024Motion-aware Latent Diffusion Models for Video Frame Interpolation
  6. 2025USV: Unified Sparsification for Accelerating Video Diffusion Models
  7. 2024IV-Mixed Sampler: Leveraging Image Diffusion Models for Enhanced Video Synthesis
  8. 2026Diff-VF: Training-free High-quality Long Video Generation via Diffusion Model