# Tracking Any Point with Frame-Event Fusion Network at High Frame Rate

arXiv: https://arxiv.org/abs/2409.11953 · HTML: https://mlsanity.com/paper/2409.11953/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Builds on (most cited first, 24 of 24)

- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement](https://mlsanity.com/paper/2306.08637/) (2023)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories](https://mlsanity.com/paper/2204.04153/) (2022)
- [The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM](https://mlsanity.com/paper/1610.08336/) (2016)
- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows](https://mlsanity.com/paper/2108.05015/) (2021)
- [Asynchronous, Photometric Feature Tracking using Events and Frames](https://mlsanity.com/paper/1807.09713/) (2018)
- [FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation](https://mlsanity.com/paper/2303.01237/) (2023)
- [Event-based High Dynamic Range Image and Very High Frame Rate Video Generation using Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1811.08230/) (2018)
- [Event-aided Direct Sparse Odometry](https://mlsanity.com/paper/2204.07640/) (2022)
- [DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Video](https://mlsanity.com/paper/2403.14548/) (2024)
- [How to Calibrate Your Event Camera](https://mlsanity.com/paper/2105.12362/) (2021)
- [Dense Continuous-Time Optical Flow from Events and Frames](https://mlsanity.com/paper/2203.13674/) (2022)
- [Frame-Event Alignment and Fusion Network for High Frame Rate Tracking](https://mlsanity.com/paper/2305.15688/) (2023)
- [RGB-Event Fusion for Moving Object Detection in Autonomous Driving](https://mlsanity.com/paper/2209.08323/) (2022)
- [FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras](https://mlsanity.com/paper/1906.10925/) (2019)
- [SplatFlow: Learning Multi-frame Optical Flow via Splatting](https://mlsanity.com/paper/2306.08887/) (2023)
- [Long-Lived Accurate Keypoints in Event Streams](https://mlsanity.com/paper/2209.10385/) (2022)
- [Detecting Line Segments in Motion-blurred Images with Events](https://mlsanity.com/paper/2211.07365/) (2022)
- [FE-DeTr: Keypoint Detection and Tracking in Low-quality Image Frames with Events](https://mlsanity.com/paper/2403.11662/) (2024)

## Built on by (most cited first, 6 of 6)

- [ETAP: Event-based Tracking of Any Point](https://mlsanity.com/paper/2412.00133/) (2024)
- [TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion](https://mlsanity.com/paper/2505.03116/) (2025)
- [MATE: Motion-Augmented Temporal Consistency for Event-based Point Tracking](https://mlsanity.com/paper/2412.01300/) (2024)
- [E-VLA: Event-Augmented Vision-Language-Action Model for Dark and Blurred Scenes](https://mlsanity.com/paper/2604.04834/) (2026)
- [TAPFormer: Robust Arbitrary Point Tracking via Transient Asynchronous Fusion of Frames and Events](https://mlsanity.com/paper/2603.04989/) (2026)
- [E-TraMamba: A New Paradigm for Efficient Long-Term 3D Feature Tracking with Event Cameras](https://mlsanity.com/paper/2607.02866/) (2026)

## Abstract (arXiv)

Tracking any point based on image frames is constrained by frame rates, leading to instability in high-speed scenarios and limited generalization in real-world applications. To overcome these limitations, we propose an image-event fusion point tracker, FE-TAP, which combines the contextual information from image frames with the high temporal resolution of events, achieving high frame rate and robust point tracking under various challenging conditions. Specifically, we designed an Evolution Fusion module (EvoFusion) to model the image generation process guided by events. This module can effectively integrate valuable information from both modalities operating at different frequencies. To achieve smoother point trajectories, we employed a transformer-based refinement strategy that updates the point's trajectories and features iteratively. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches, particularly improving expected feature age by 24% on EDS datasets. Finally, we qualitatively validated the robustness of our algorithm in real driving scenarios using our custom-designed high-resolution image-event synchronization device. Our source code will be released at https://github.com/ljx1002/FE-TAP.