# Towards General Multimodal Visual Tracking

arXiv: https://arxiv.org/abs/2503.11218 · HTML: https://mlsanity.com/paper/2503.11218/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 53 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Builds on (most cited first, 30 of 40)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022)
- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [VM-UNet: Vision Mamba UNet for Medical Image Segmentation](https://mlsanity.com/paper/2402.02491/) (2024)
- [LocalMamba: Visual State Space Model with Windowed Selective Scan](https://mlsanity.com/paper/2403.09338/) (2024)
- [Visual Prompt Multi-Modal Tracking](https://mlsanity.com/paper/2303.10826/) (2023)
- [PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition](https://mlsanity.com/paper/2403.17695/) (2024)
- [VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows](https://mlsanity.com/paper/2108.05015/) (2021)
- [Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark](https://mlsanity.com/paper/2103.16746/) (2021)
- [RGB-T Object Tracking:Benchmark and Baseline](https://mlsanity.com/paper/1805.08982/) (2018)
- [ChangeMamba: Remote Sensing Change Detection With Spatiotemporal State Space Model](https://mlsanity.com/paper/2404.03425/) (2024)
- [Transforming Model Prediction for Tracking](https://mlsanity.com/paper/2203.11192/) (2022)
- [LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking](https://mlsanity.com/paper/2104.13202/) (2021)
- [RSMamba: Remote Sensing Image Classification with State Space Model](https://mlsanity.com/paper/2403.19654/) (2024)
- [Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline](https://mlsanity.com/paper/2204.04120/) (2022)
- [Object Tracking by Jointly Exploiting Frame and Event Domain](https://mlsanity.com/paper/2109.09052/) (2021)
- [Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline](https://mlsanity.com/paper/2309.14611/) (2023)
- [Single-Model and Any-Modality for Video Object Tracking](https://mlsanity.com/paper/2311.15851/) (2023)
- [Towards Unified Token Learning for Vision-Language Tracking](https://mlsanity.com/paper/2308.14103/) (2023)
- [MiM-ISTD: Mamba-in-Mamba for Efficient Infrared Small Target Detection](https://mlsanity.com/paper/2403.02148/) (2024)
- [Prompting for Multi-Modal Tracking](https://mlsanity.com/paper/2207.14571/) (2022)
- [Joint Visual Grounding and Tracking with Natural Language Specification](https://mlsanity.com/paper/2303.12027/) (2023)
- [Revisiting Color-Event based Tracking: A Unified Network, Dataset, and Metric](https://mlsanity.com/paper/2211.11010/) (2022)
- [Divert More Attention to Vision-Language Tracking](https://mlsanity.com/paper/2207.01076/) (2022)

## Built on by (most cited first, 1 of 1)

- [Omni Survey for Multimodality Analysis in Visual Object Tracking](https://mlsanity.com/paper/2508.13000/) (2025)

## Abstract (arXiv)

Existing multimodal tracking studies focus on bi-modal scenarios such as RGB-Thermal, RGB-Event, and RGB-Language. Although promising tracking performance is achieved through leveraging complementary cues from different sources, it remains challenging in complex scenes due to the limitations of bi-modal scenarios. In this work, we introduce a general multimodal visual tracking task that fully exploits the advantages of four modalities, including RGB, thermal infrared, event, and language, for robust tracking under challenging conditions. To provide a comprehensive evaluation platform for general multimodal visual tracking, we construct QuadTrack600, a large-scale, high-quality benchmark comprising 600 video sequences (totaling 384.7K high-resolution (640x480) frame groups). In each frame group, all four modalities are spatially aligned and meticulously annotated with bounding boxes, while 21 sequence-level challenge attributes are provided for detailed performance analysis. Despite quad-modal data provides richer information, the differences in information quantity among modalities and the computational burden from four modalities are two challenging issues in fusing four modalities. To handle these issues, we propose a novel approach called QuadFusion, which incorporates an efficient Multiscale Fusion Mamba with four different scanning scales to achieve sufficient interactions of the four modalities while overcoming the exponential computational burden, for general multimodal visual tracking. Extensive experiments on the QuadTrack600 dataset and three bi-modal tracking datasets, including LasHeR, VisEvent, and TNL2K, validate the effectiveness of our QuadFusion.