# TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion

arXiv: https://arxiv.org/abs/2505.03116 · HTML: https://mlsanity.com/paper/2505.03116/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 62 references that are in the map.
- First posted on arXiv 6 May 2025.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement](https://mlsanity.com/paper/2306.08637/) (2023)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking](https://mlsanity.com/paper/2307.15055/) (2023)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [Video Frame Interpolation via Adaptive Separable Convolution](https://mlsanity.com/paper/1708.01692/) (2017)
- [Real-Time Intermediate Flow Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2011.06294/) (2020)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [Video to Events: Recycling Video Datasets for Event Cameras](https://mlsanity.com/paper/1912.03095/) (2019)
- [Video Frame Interpolation via Adaptive Convolution](https://mlsanity.com/paper/1703.07514/) (2017)
- [E-RAFT: Dense Optical Flow from Event Cameras](https://mlsanity.com/paper/2108.10552/) (2021)
- [TimeLens: Event-based Video Frame Interpolation](https://mlsanity.com/paper/2106.07286/) (2021)
- [Quadratic video interpolation](https://mlsanity.com/paper/1911.00627/) (2019)
- [Video Frame Interpolation Transformer](https://mlsanity.com/paper/2111.13817/) (2021)
- [Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation](https://mlsanity.com/paper/2003.13045/) (2020)
- [Asynchronous, Photometric Feature Tracking using Events and Frames](https://mlsanity.com/paper/1807.09713/) (2018)
- [BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolation](https://mlsanity.com/paper/2007.12622/) (2020)
- [Time Lens++: Event-based Frame Interpolation with Parametric Non-linear Flow and Multi-scale Fusion](https://mlsanity.com/paper/2203.17191/) (2022)
- [Local All-Pair Correspondence for Point Tracking](https://mlsanity.com/paper/2407.15420/) (2024)

## Built on by (most cited first, 9 of 9)

- [EventDiff: A Unified and Efficient Diffusion Model Framework for Event-based Video Frame Interpolation](https://mlsanity.com/paper/2505.08235/) (2025)
- [Event Camera Guided Visual Media Restoration & 3D Reconstruction: A Survey](https://mlsanity.com/paper/2509.09971/) (2025)
- [Seeing the Unseen: Zooming in the Dark with Event Cameras](https://mlsanity.com/paper/2601.02206/) (2026)
- [Towards Robust and Generalizable Continuous Space-Time Video Super-Resolution with Events](https://mlsanity.com/paper/2510.03833/) (2025)
- [UniE2F: A Unified Diffusion Framework for Event-to-Frame Reconstruction with Video Foundation Models](https://mlsanity.com/paper/2602.19202/) (2026)
- [TETO: Tracking Events with Teacher Observation for Motion Estimation and Frame Interpolation](https://mlsanity.com/paper/2603.23487/) (2026)
- [High-Quality and Efficient Turbulence Mitigation with Events](https://mlsanity.com/paper/2603.20708/) (2026)
- [UniRED: Unified RGB-D Video Frame Interpolation with Event Guidance](https://mlsanity.com/paper/2606.24282/) (2026)
- [GoStop: Reinforcement Learning for Adaptive Temporal Aggregation in Event-Based Feature Tracking](https://mlsanity.com/paper/2607.15699/) (2026)

## Abstract (arXiv)

Video frame interpolation (VFI) that leverages the bio-inspired event cameras as guidance has recently shown better performance and memory efficiency than the frame-based methods, thanks to the event cameras' advantages, such as high temporal resolution. A hurdle for event-based VFI is how to effectively deal with non-linear motion, caused by the dynamic changes in motion direction and speed within the scene. Existing methods either use events to estimate sparse optical flow or fuse events with image features to estimate dense optical flow. Unfortunately, motion errors often degrade the VFI quality as the continuous motion cues from events do not align with the dense spatial information of images in the temporal dimension. In this paper, we find that object motion is continuous in space, tracking local regions over continuous time enables more accurate identification of spatiotemporal feature correlations. In light of this, we propose a novel continuous point tracking-based VFI framework, named TimeTracker. Specifically, we first design a Scene-Aware Region Segmentation (SARS) module to divide the scene into similar patches. Then, a Continuous Trajectory guided Motion Estimation (CTME) module is proposed to track the continuous motion trajectory of each patch through events. Finally, intermediate frames at any given time are generated through global motion optimization and frame refinement. Moreover, we collect a real-world dataset that features fast non-linear motion. Extensive experiments show that our method outperforms prior arts in both motion estimation and frame interpolation quality.