mlsanityData to 7 Oct 2026

LDMVFI: Video Frame Interpolation with Latent Diffusion Models

arXiv 2303.09508 ↗First posted 16 Mar 2023eess.IV cs.CVcited by 88

On the map: Diffusion models → Video generation

Abstract (arXiv)

Existing works on video frame interpolation (VFI) mostly employ deep neural networks that are trained by minimizing the L1, L2, or deep feature space distance (e.g. VGG loss) between their outputs and ground-truth frames. However, recent works have shown that these metrics are poor indicators of perceptual VFI quality. Towards developing perceptually-oriented VFI methods, in this work we propose latent diffusion model-based VFI, LDMVFI. This approaches the VFI problem from a generative perspective by formulating it as a conditional generation problem. As the first effort to address VFI using latent diffusion models, we rigorously benchmark our method on common test sets used in the existing VFI literature. Our quantitative experiments and user study indicate that LDMVFI is able to interpolate video content with favorable perceptual quality compared to the state of the art, even in the high-resolution regime. Our code is available at https://github.com/danier97/LDMVFI.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 61 of its 84 references that are in the map. First posted on arXiv 16 Mar 2023.

Builds on

61 of 84 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
  2. 2022Video Diffusion Models7 Apr

Other references · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2020Denoising Diffusion Implicit Modelscited by 9,655
  11. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  12. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
38 more
  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  2. 2022Generative Adversarial Networkscited by 5,603
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  7. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  8. 2017Deformable Convolutional Networkscited by 1,481
  9. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  10. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  11. 2017Video Enhancement with Task-Oriented Flowcited by 612
  12. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  13. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  14. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  15. 2017Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolationcited by 321
  16. 2020Softmax Splatting for Video Frame Interpolationcited by 276
  17. 2019Depth-Aware Video Frame Interpolationcited by 232
  18. 2017Video Frame Interpolation via Adaptive Separable Convolutioncited by 211
  19. 2020Real-Time Intermediate Flow Estimation for Video Frame Interpolationcited by 211
  20. 2017Video Frame Synthesis using Deep Voxel Flowcited by 209
  21. 2015DeepStereo: Learning to Predict New Views from the World's Imagerycited by 185
  22. 2017Video Frame Interpolation via Adaptive Convolutioncited by 158
  23. 2018Context-aware Synthesis for Video Frame Interpolationcited by 153
  24. 2022FILM: Frame Interpolation for Large Motioncited by 149
  25. 2018Video Compression through Image Interpolationcited by 143
  26. 2019Quadratic video interpolationcited by 143
  27. 2019AdaCoF: Adaptive Collaboration of Flows for Video Frame Interpolationcited by 133
  28. 2021XVFI: eXtreme Video Frame Interpolationcited by 123
  29. 2021Video Frame Interpolation Transformercited by 122
  30. 2020BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolationcited by 117
  31. 2018MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancementcited by 111
  32. 2022IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolationcited by 107
  33. 2021Asymmetric Bilateral Motion Estimation for Video Frame Interpolationcited by 101
  34. 2020FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolationcited by 97
  35. 2021Deep Animation Video Interpolation in the Wildcited by 65
  36. 2020Multiple Video Frame Interpolation via Enhanced Deformable Separable Convolutioncited by 62
  37. 2021CDFI: Compression-Driven Network Design for Frame Interpolationcited by 55
  38. 2020BVI-DVC: A Training Database for Deep Video Compressioncited by 47

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 28; 2025: 33; 2026: 21.

88 papers.

Sort:

By area:

  1. 2023A Survey on Video Diffusion Modelscited by 203
  2. 2024Diffusion Models in Low-Level Vision: A Surveycited by 73
  3. 2024ToonCrafter: Generative Cartoon Interpolationcited by 68
  4. 2024Video Interpolation with Diffusion Modelscited by 67
  5. 2024Controlling Space and Time with Diffusion Modelscited by 65
  6. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49
  7. 2024MagicVideo-V2: Multi-Stage High-Aesthetic Video Generationcited by 48
  8. 2023Norm-guided latent space exploration for text-to-image generationcited by 43
  9. 2024TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolationcited by 32
  10. 2024Framer: Interactive Frame Interpolationcited by 24
  11. 2024Perception-Oriented Video Frame Interpolation via Asymmetric Blendingcited by 23
  12. 2024Generative Inbetweening through Frame-wise Conditions-Driven Video Generationcited by 23
38 more
  1. 2024AniDoc: Animation Creation Made Easiercited by 22
  2. 2024ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Samplercited by 19
  3. 2025EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolationcited by 19
  4. 2024Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep Approachcited by 18
  5. 2024Sakuga-42M Dataset: Scaling Up Cartoon Researchcited by 17
  6. 2024DreamMover: Leveraging the Prior of Diffusion Models for Image Interpolation with Large Motioncited by 16
  7. 2024Motion-aware Latent Diffusion Models for Video Frame Interpolationcited by 14
  8. 2024UniPaint: Unified Space-time Video Inpainting via Mixture-of-Expertscited by 13
  9. 2025Motion-Aware Generative Frame Interpolationcited by 13
  10. 2024Disentangled Motion Modeling for Video Frame Interpolationcited by 11
  11. 2025GIViC: Generative Implicit Video Compressioncited by 9
  12. 2025AceVFI: A Comprehensive Survey of Advances in Video Frame Interpolationcited by 8
  13. 2024KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photoscited by 7
  14. 2025Hierarchical Flow Diffusion for Efficient Frame Interpolationcited by 7
  15. 2025In-2-4D: Inbetweening from Two Single-View Images to 4D Generationcited by 7
  16. 2025StreamSplat: Towards Online Dynamic 3D Reconstruction from Uncalibrated Video Streamscited by 7
  17. 2025TLB-VFI: Temporal-Aware Latent Brownian Bridge Diffusion for Video Frame Interpolationcited by 7
  18. 2025EGVD: Event-Guided Video Diffusion Model for Physically Realistic Large-Motion Frame Interpolationcited by 6
  19. 2023FLAIR: A Conditional Diffusion Framework with Applications to Face Video Restorationcited by 5
  20. 2024ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video Generationcited by 5
  21. 2025Video Motion Graphscited by 5
  22. 2025FB-Diff: Fourier Basis-guided Diffusion for Temporal Interpolation of 4D Medical Imagingcited by 5
  23. 2026Motion Prior Distillation in Time Reversal Sampling for Generative Inbetweeningcited by 5
  24. 2024VegeDiff: Latent Diffusion Model for Geospatial Vegetation Forecastingcited by 4
  25. 2025Semantic-Aware Adaptive Video Streaming Using Latent Diffusion Models for Wireless Networkscited by 4
  26. 2025AnyMoLe: Any Character Motion In-betweening Leveraging Video Diffusion Modelscited by 4
  27. 2025VideoCanvas: Unified Video Completion from Arbitrary Spatiotemporal Patches via In-Context Conditioningcited by 4
  28. 2026Towards Holistic Modeling for Video Frame Interpolation with Auto-regressive Diffusion Transformerscited by 4
  29. 2026MRI Contrast Enhancement Kinetics World Modelcited by 4
  30. 2026UniTemp: Unlocking Video Generation in Any Temporal Order via Bidirectional Distillationcited by 4
  31. 2025EventDiff: A Unified and Efficient Diffusion Model Framework for Event-based Video Frame Interpolationcited by 3
  32. 2025Temporal Differential Fields for 4D Motion Modeling via Image-to-Video Synthesiscited by 3
  33. 2025EF-VI: Enhancing End-Frame Injection for Video Inbetweeningcited by 3
  34. 2024RMT-BVQA: Recurrent Memory Transformer-based Blind Video Quality Assessment for Enhanced Video Contentcited by 2
  35. 2024Survey on Visual Signal Coding and Processing with Generative Models: Technologies, Standards and Optimizationcited by 2
  36. 2024Artificial Intelligence for Biomedical Video Generationcited by 2
  37. 2024Reversing the Damage: A QP-Aware Transformer-Diffusion Approach for 8K Video Restoration under Codec Compressioncited by 2
  38. 2025Time-adaptive Video Frame Interpolation based on Residual Diffusioncited by 2

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Frame Interpolation with Consecutive Brownian Bridge Diffusion
  2. 2024Repurposing Pre-trained Video Diffusion Models for Event-based Video Interpolation
  3. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
  4. 2024VFIMamba: Video Frame Interpolation with State Space Models
  5. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  6. 2023Video Probabilistic Diffusion Models in Projected Latent Space
  7. 2022Diffusion Models for Video Prediction and Infilling
  8. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation