# TimeReplayer: Unlocking the Potential of Event Cameras for Video Interpolation

arXiv: https://arxiv.org/abs/2203.13859 · HTML: https://mlsanity.com/paper/2203.13859/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 25 Mar 2022.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [Video Frame Interpolation via Adaptive Separable Convolution](https://mlsanity.com/paper/1708.01692/) (2017)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [TimeLens: Event-based Video Frame Interpolation](https://mlsanity.com/paper/2106.07286/) (2021)
- [Quadratic video interpolation](https://mlsanity.com/paper/1911.00627/) (2019)
- [Bringing a Blurry Frame Alive at High Frame-Rate with an Event Camera](https://mlsanity.com/paper/1811.10180/) (2018)
- [BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolation](https://mlsanity.com/paper/2007.12622/) (2020)
- [MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement](https://mlsanity.com/paper/1810.08768/) (2018)
- [Learning Dense Correspondence via 3D-guided Cycle Consistency](https://mlsanity.com/paper/1604.05383/) (2016)
- [DDD17: End-To-End DAVIS Driving Dataset](https://mlsanity.com/paper/1711.01458/) (2017)
- [Learning Event-Based Motion Deblurring](https://mlsanity.com/paper/2004.05794/) (2020)
- [Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation](https://mlsanity.com/paper/1607.06283/) (2016)
- [Enhanced Quadratic Video Interpolation](https://mlsanity.com/paper/2009.04642/) (2020)
- [Unsupervised Video Interpolation Using Cycle Consistency](https://mlsanity.com/paper/1906.05928/) (2019)
- [Comparing SNNs and RNNs on Neuromorphic Vision Datasets: Similarities and Differences](https://mlsanity.com/paper/2005.02183/) (2020)
- [DashNet: A Hybrid Artificial and Spiking Neural Network for High-speed Object Tracking](https://mlsanity.com/paper/1909.12942/) (2019)

## Built on by (most cited first, 30 of 37)

- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [Recent Event Camera Innovations: A Survey](https://mlsanity.com/paper/2408.13627/) (2024)
- [Low-Light Video Enhancement with Synthetic Event Guidance](https://mlsanity.com/paper/2208.11014/) (2022)
- [A Unified Framework for Event-based Frame Interpolation with Ad-hoc Deblurring in the Wild](https://mlsanity.com/paper/2301.05191/) (2023)
- [Event-Based Video Frame Interpolation With Cross-Modal Asymmetric Bidirectional Motion Fields](https://mlsanity.com/paper/2502.13716/) (2025)
- [Video Interpolation by Event-driven Anisotropic Adjustment of Optical Flow](https://mlsanity.com/paper/2208.09127/) (2022)
- [Data-Efficient Unsupervised Interpolation Without Any Intermediate Frame for 4D Medical Images](https://mlsanity.com/paper/2404.01464/) (2024)
- [SportsSloMo: A New Benchmark and Baselines for Human-centric Video Frame Interpolation](https://mlsanity.com/paper/2308.16876/) (2023)
- [Video Frame Interpolation with Stereo Event and Intensity Camera](https://mlsanity.com/paper/2307.08228/) (2023)
- [TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion](https://mlsanity.com/paper/2505.03116/) (2025)
- [AceVFI: A Comprehensive Survey of Advances in Video Frame Interpolation](https://mlsanity.com/paper/2506.01061/) (2025)
- [BVI-VFI: A Video Quality Database for Video Frame Interpolation](https://mlsanity.com/paper/2210.00823/) (2022)
- [Event-Based Motion Magnification](https://mlsanity.com/paper/2402.11957/) (2024)
- [E-4DGS: High-Fidelity Dynamic Reconstruction from the Multi-view Event Cameras](https://mlsanity.com/paper/2508.09912/) (2025)
- [HR-INR: Continuous Space-Time Video Super-Resolution via Event Camera](https://mlsanity.com/paper/2405.13389/) (2024)
- [Finding Meaning in Points: Weakly Supervised Semantic Segmentation for Event Cameras](https://mlsanity.com/paper/2407.11216/) (2024)
- [GS2E: Gaussian Splatting is an Effective Data Generator for Event Stream Generation](https://mlsanity.com/paper/2505.15287/) (2025)
- [Self-supervised Learning of Event-guided Video Frame Interpolation for Rolling Shutter Frames](https://mlsanity.com/paper/2306.15507/) (2023)
- [Neural Image Re-Exposure](https://mlsanity.com/paper/2305.13593/) (2023)
- [CrossZoom: Simultaneously Motion Deblurring and Event Super-Resolving](https://mlsanity.com/paper/2309.16949/) (2023)
- [Revisiting Event-based Video Frame Interpolation](https://mlsanity.com/paper/2307.12558/) (2023)
- [Event-based Video Frame Interpolation with Edge Guided Motion Refinement](https://mlsanity.com/paper/2404.18156/) (2024)
- [Event Camera Guided Visual Media Restoration & 3D Reconstruction: A Survey](https://mlsanity.com/paper/2509.09971/) (2025)
- [Recovering Continuous Scene Dynamics from A Single Blurry Image with Events](https://mlsanity.com/paper/2304.02695/) (2023)
- [Self-Supervised Scene Dynamic Recovery from Rolling Shutter Images and Events](https://mlsanity.com/paper/2304.06930/) (2023)
- [IDO-VFI: Identifying Dynamics via Optical Flow Guidance for Video Frame Interpolation with Events](https://mlsanity.com/paper/2305.10198/) (2023)
- [Towards Robust and Generalizable Continuous Space-Time Video Super-Resolution with Events](https://mlsanity.com/paper/2510.03833/) (2025)
- [UniE2F: A Unified Diffusion Framework for Event-to-Frame Reconstruction with Video Foundation Models](https://mlsanity.com/paper/2602.19202/) (2026)
- [From Events to Enhancement: A Survey on Event-Based Imaging Technologies](https://mlsanity.com/paper/2505.05488/) (2025)
- [EVDI++: Event-based Video Deblurring and Interpolation via Self-Supervised Learning](https://mlsanity.com/paper/2509.08260/) (2025)

## Abstract (arXiv)

Recording fast motion in a high FPS (frame-per-second) requires expensive high-speed cameras. As an alternative, interpolating low-FPS videos from commodity cameras has attracted significant attention. If only low-FPS videos are available, motion assumptions (linear or quadratic) are necessary to infer intermediate frames, which fail to model complex motions. Event camera, a new camera with pixels producing events of brightness change at the temporal resolution of μ s (10-6 second ), is a game-changing device to enable video interpolation at the presence of arbitrarily complex motion. Since event camera is a novel sensor, its potential has not been fulfilled due to the lack of processing algorithms. The pioneering work Time Lens introduced event cameras to video interpolation by designing optical devices to collect a large amount of paired training data of high-speed frames and events, which is too costly to scale. To fully unlock the potential of event cameras, this paper proposes a novel TimeReplayer algorithm to interpolate videos captured by commodity cameras with events. It is trained in an unsupervised cycle-consistent style, canceling the necessity of high-speed training data and bringing the additional ability of video extrapolation. Its state-of-the-art results and demo videos in supplementary reveal the promising future of event-based vision.