# Repurposing Pre-trained Video Diffusion Models for Event-based Video Interpolation

arXiv: https://arxiv.org/abs/2412.07761 · HTML: https://mlsanity.com/paper/2412.07761/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 40 of its 55 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 40)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [Real-Time Intermediate Flow Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2011.06294/) (2020)
- [MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance](https://mlsanity.com/paper/2406.19680/) (2024)
- [TimeLens: Event-based Video Frame Interpolation](https://mlsanity.com/paper/2106.07286/) (2021)
- [Event-based Moving Object Detection and Tracking](https://mlsanity.com/paper/1803.04523/) (2018)
- [Video Frame Interpolation Transformer](https://mlsanity.com/paper/2111.13817/) (2021)
- [Time Lens++: Event-based Frame Interpolation with Parametric Non-linear Flow and Multi-scale Fusion](https://mlsanity.com/paper/2203.17191/) (2022)
- [IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation](https://mlsanity.com/paper/2205.14620/) (2022)
- [Asymmetric Bilateral Motion Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2108.06815/) (2021)
- [Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation](https://mlsanity.com/paper/2303.00440/) (2023)

## Built on by (most cited first, 23 of 23)

- [First Frame Is the Place to Go for Video Content Customization](https://mlsanity.com/paper/2511.15700/) (2025)
- [AceVFI: A Comprehensive Survey of Advances in Video Frame Interpolation](https://mlsanity.com/paper/2506.01061/) (2025)
- [EGVD: Event-Guided Video Diffusion Model for Physically Realistic Large-Motion Frame Interpolation](https://mlsanity.com/paper/2503.20268/) (2025)
- [Learning Normal Flow Directly From Event Neighborhoods](https://mlsanity.com/paper/2412.11284/) (2024)
- [Parametric Shadow Control for Portrait Generation in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2503.21943/) (2025)
- [From Inpainting to Layer Decomposition: Repurposing Generative Inpainting Models for Image Layer Decomposition](https://mlsanity.com/paper/2511.20996/) (2025)
- [EventDiff: A Unified and Efficient Diffusion Model Framework for Event-based Video Frame Interpolation](https://mlsanity.com/paper/2505.08235/) (2025)
- [Event Camera Guided Visual Media Restoration & 3D Reconstruction: A Survey](https://mlsanity.com/paper/2509.09971/) (2025)
- [Time-Correlated Video Bridge Matching](https://mlsanity.com/paper/2510.12453/) (2025)
- [Event6D: Event-based Novel Object 6D Pose Tracking](https://mlsanity.com/paper/2603.28045/) (2026)
- [Towards Robust and Generalizable Continuous Space-Time Video Super-Resolution with Events](https://mlsanity.com/paper/2510.03833/) (2025)
- [DESSERT: Diffusion-based Event-driven Single-frame Synthesis via Residual Training](https://mlsanity.com/paper/2512.17323/) (2025)
- [UniE2F: A Unified Diffusion Framework for Event-to-Frame Reconstruction with Video Foundation Models](https://mlsanity.com/paper/2602.19202/) (2026)
- [TETO: Tracking Events with Teacher Observation for Motion Estimation and Frame Interpolation](https://mlsanity.com/paper/2603.23487/) (2026)
- [IE2Video: Adapting Pretrained Diffusion Models for Event-Based Video Reconstruction](https://mlsanity.com/paper/2512.05240/) (2025)
- [FC-VFI: Faithful and Consistent Video Frame Interpolation for High-FPS Slow Motion Video Generation](https://mlsanity.com/paper/2603.04899/) (2026)
- [Interp3R: Continuous-time 3D Geometry Estimation with Frames and Events](https://mlsanity.com/paper/2603.14528/) (2026)
- [Single-Shot HDR Recovery via a Video Diffusion Prior](https://mlsanity.com/paper/2605.11628/) (2026)
- [LongE2V: Long-Horizon Event-based Video Reconstruction, Prediction, and Frame Interpolation with Video Diffusion Models](https://mlsanity.com/paper/2607.08770/) (2026)
- [Engram-E2VID: Reference-Based Event-to-Video Reconstruction via Generative Activation of Appearance Engrams](https://mlsanity.com/paper/2608.05728/) (2026)
- [Bridging Event Streams and DiT: Event-Guided Video Frame Interpolation](https://mlsanity.com/paper/2608.10479/) (2026)
- [Adapting Vision Foundation Models to Acoustics for Pose-Free 3D Sonar Reconstruction](https://mlsanity.com/paper/2609.06261/) (2026)
- [Unifying Video Tasks via Spatiotemporal Analogy](https://mlsanity.com/paper/2609.33935/) (2026)

## Abstract (arXiv)

Video Frame Interpolation aims to recover realistic missing frames between observed frames, generating a high-frame-rate video from a low-frame-rate video. However, without additional guidance, the large motion between frames makes this problem ill-posed. Event-based Video Frame Interpolation (EVFI) addresses this challenge by using sparse, high-temporal-resolution event measurements as motion guidance. This guidance allows EVFI methods to significantly outperform frame-only methods. However, to date, EVFI methods have relied on a limited set of paired event-frame training data, severely limiting their performance and generalization capabilities. In this work, we overcome the limited data challenge by adapting pre-trained video diffusion models trained on internet-scale datasets to EVFI. We experimentally validate our approach on real-world EVFI datasets, including a new one that we introduce. Our method outperforms existing methods and generalizes across cameras far better than existing approaches.