# VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows

arXiv: https://arxiv.org/abs/2108.05015 · HTML: https://mlsanity.com/paper/2108.05015/

## Key facts

- Cited by 129 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 128 references that are in the map.
- First posted on arXiv 11 Aug 2021.

## Builds on (most cited first, 30 of 65)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [Probabilistic Regression for Visual Tracking](https://mlsanity.com/paper/2003.12565/) (2020)
- [Ocean: Object-aware Anchor-free Tracking](https://mlsanity.com/paper/2006.10721/) (2020)
- [Learning Multi-Domain Convolutional Neural Networks for Visual Tracking](https://mlsanity.com/paper/1510.07945/) (2015)
- [Siamese Box Adaptive Network for Visual Tracking](https://mlsanity.com/paper/2003.06761/) (2020)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)
- [SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelines](https://mlsanity.com/paper/1911.06188/) (2019)
- [Siam R-CNN: Visual Tracking by Re-Detection](https://mlsanity.com/paper/1911.12836/) (2019)
- [Deeper and Wider Siamese Networks for Real-Time Visual Tracking](https://mlsanity.com/paper/1901.01660/) (2019)
- [SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Tracking](https://mlsanity.com/paper/1911.07241/) (2019)
- [Need for Speed: A Benchmark for Higher Frame Rate Object Tracking](https://mlsanity.com/paper/1703.05884/) (2017)
- [Event-based Moving Object Detection and Tracking](https://mlsanity.com/paper/1803.04523/) (2018)
- [Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark](https://mlsanity.com/paper/2103.16746/) (2021)
- [RGB-T Object Tracking:Benchmark and Baseline](https://mlsanity.com/paper/1805.08982/) (2018)
- [Know Your Surroundings: Exploiting Scene Information for Object Tracking](https://mlsanity.com/paper/2003.11014/) (2020)
- [Asynchronous, Photometric Feature Tracking using Events and Frames](https://mlsanity.com/paper/1807.09713/) (2018)
- [High-Performance Long-Term Tracking with Meta-Updater](https://mlsanity.com/paper/2004.00305/) (2020)
- [Unsupervised Deep Tracking](https://mlsanity.com/paper/1904.01828/) (2019)

## Built on by (most cited first, 30 of 129)

- [Visual Prompt Multi-Modal Tracking](https://mlsanity.com/paper/2303.10826/) (2023)
- [Global Structure-Aware Diffusion Process for Low-Light Image Enhancement](https://mlsanity.com/paper/2310.17577/) (2023)
- [Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline](https://mlsanity.com/paper/2309.14611/) (2023)
- [Single-Model and Any-Modality for Video Object Tracking](https://mlsanity.com/paper/2311.15851/) (2023)
- [OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning](https://mlsanity.com/paper/2403.09634/) (2024)
- [Event-based Video Reconstruction via Potential-assisted Spiking Neural Network](https://mlsanity.com/paper/2201.10943/) (2022)
- [Prompting for Multi-Modal Tracking](https://mlsanity.com/paper/2207.14571/) (2022)
- [Revisiting Color-Event based Tracking: A Unified Network, Dataset, and Metric](https://mlsanity.com/paper/2211.11010/) (2022)
- [HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors](https://mlsanity.com/paper/2211.09648/) (2022)
- [SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking](https://mlsanity.com/paper/2403.16002/) (2024)
- [Frame-Event Alignment and Fusion Network for High Frame Rate Tracking](https://mlsanity.com/paper/2305.15688/) (2023)
- [SODFormer: Streaming Object Detection with Transformer Using Events and Frames](https://mlsanity.com/paper/2308.04047/) (2023)
- [Towards Universal Modal Tracking with Online Dense Temporal Token Learning](https://mlsanity.com/paper/2507.20177/) (2025)
- [Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusion](https://mlsanity.com/paper/2312.10692/) (2023)
- [Mamba-FETrack: Frame-Event Tracking via State Space Model](https://mlsanity.com/paper/2404.18174/) (2024)
- [Unified Sequence-to-Sequence Learning for Single- and Multi-Modal Visual Object Tracking](https://mlsanity.com/paper/2304.14394/) (2023)
- [XTrack: Multimodal Training Boosts RGB-X Video Object Trackers](https://mlsanity.com/paper/2405.17773/) (2024)
- [SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition](https://mlsanity.com/paper/2308.04369/) (2023)
- [EventAid: Benchmarking Event-aided Image/Video Enhancement Algorithms with Real-captured Hybrid Dataset](https://mlsanity.com/paper/2312.08220/) (2023)
- [Learning Spatial-Frequency Transformer for Visual Object Tracking](https://mlsanity.com/paper/2208.08829/) (2022)
- [CRSOT: Cross-Resolution Object Tracking using Unaligned Frame and Event Cameras](https://mlsanity.com/paper/2401.02826/) (2024)
- [DailyDVS-200: A Comprehensive Benchmark Dataset for Event-Based Action Recognition](https://mlsanity.com/paper/2407.05106/) (2024)
- [Enhancing Traffic Object Detection in Variable Illumination with RGB-Event Fusion](https://mlsanity.com/paper/2311.00436/) (2023)
- [FRED: The Florence RGB-Event Drone Dataset](https://mlsanity.com/paper/2506.05163/) (2025)
- [What You Have is What You Track: Adaptive and Robust Multimodal Tracking](https://mlsanity.com/paper/2507.05899/) (2025)
- [Neuromorphic Drone Detection: an Event-RGB Multimodal Approach](https://mlsanity.com/paper/2409.16099/) (2024)
- [Event-based Tiny Object Detection: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2506.23575/) (2025)
- [SRFNet: Monocular Depth Estimation with Fine-grained Structure via Spatial Reliability-oriented Fusion of Frames and Events](https://mlsanity.com/paper/2309.12842/) (2023)
- [MambaEVT: Event Stream based Visual Object Tracking using State Space Model](https://mlsanity.com/paper/2408.10487/) (2024)
- [E-Motion: Future Motion Simulation via Event Sequence Diffusion](https://mlsanity.com/paper/2410.08649/) (2024)

## Abstract (arXiv)

Different from visible cameras which record intensity images frame by frame, the biologically inspired event camera produces a stream of asynchronous and sparse events with much lower latency. In practice, visible cameras can better perceive texture details and slow motion, while event cameras can be free from motion blurs and have a larger dynamic range which enables them to work well under fast motion and low illumination. Therefore, the two sensors can cooperate with each other to achieve more reliable object tracking. In this work, we propose a large-scale Visible-Event benchmark (termed VisEvent) due to the lack of a realistic and scaled dataset for this task. Our dataset consists of 820 video pairs captured under low illumination, high speed, and background clutter scenarios, and it is divided into a training and a testing subset, each of which contains 500 and 320 videos, respectively. Based on VisEvent, we transform the event flows into event images and construct more than 30 baseline methods by extending current single-modality trackers into dual-modality versions. More importantly, we further build a simple but effective tracking algorithm by proposing a cross-modality transformer, to achieve more effective feature fusion between visible and event data. Extensive experiments on the proposed VisEvent dataset, FE108, COESOT, and two simulated datasets (i.e., OTB-DVS and VOT-DVS), validated the effectiveness of our model. The dataset and source code have been released on: urlhttps://github.com/wangxiao5791509/VisEventSOTBenchmark.