# Cross-Modal Object Tracking: Modality-Aware Representations and A Unified Benchmark

arXiv: https://arxiv.org/abs/2111.04264 · HTML: https://mlsanity.com/paper/2111.04264/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 29 references that are in the map.
- First posted on arXiv 8 Nov 2021.

## Builds on (most cited first, 24 of 24)

- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [Ocean: Object-aware Anchor-free Tracking](https://mlsanity.com/paper/2006.10721/) (2020)
- [Learning Multi-Domain Convolutional Neural Networks for Visual Tracking](https://mlsanity.com/paper/1510.07945/) (2015)
- [Siamese Box Adaptive Network for Visual Tracking](https://mlsanity.com/paper/2003.06761/) (2020)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)
- [Deeper and Wider Siamese Networks for Real-Time Visual Tracking](https://mlsanity.com/paper/1901.01660/) (2019)
- [RGB-T Object Tracking:Benchmark and Baseline](https://mlsanity.com/paper/1805.08982/) (2018)
- [High-Performance Long-Term Tracking with Meta-Updater](https://mlsanity.com/paper/2004.00305/) (2020)
- [VITAL: VIsual Tracking via Adversarial Learning](https://mlsanity.com/paper/1804.04273/) (2018)
- [GlobalTrack: A Simple and Strong Baseline for Long-term Tracking](https://mlsanity.com/paper/1912.08531/) (2019)
- [Challenge-Aware RGBT Tracking](https://mlsanity.com/paper/2007.13143/) (2020)
- [GradNet: Gradient-Guided Network for Visual Object Tracking](https://mlsanity.com/paper/1909.06800/) (2019)
- [Real-Time MDNet](https://mlsanity.com/paper/1808.08834/) (2018)
- ['Skimming-Perusal' Tracking: A Framework for Real-Time and Robust Long-term Tracking](https://mlsanity.com/paper/1909.01840/) (2019)
- [Visual Tracking by TridentAlign and Context Embedding](https://mlsanity.com/paper/2007.06887/) (2020)

## Built on by (most cited first, 7 of 7)

- [Modality-missing RGBT Tracking: Invertible Prompt Learning and High-quality Benchmarks](https://mlsanity.com/paper/2312.16244/) (2023)
- [Cross-Modal Object Tracking via Modality-Aware Fusion Network and A Large-Scale Dataset](https://mlsanity.com/paper/2312.14446/) (2023)
- [Prototype-based Cross-Modal Object Tracking](https://mlsanity.com/paper/2312.14471/) (2023)
- [Omni Survey for Multimodality Analysis in Visual Object Tracking](https://mlsanity.com/paper/2508.13000/) (2025)
- [SwiTrack: Tri-State Switch for Cross-Modal Object Tracking](https://mlsanity.com/paper/2511.16227/) (2025)
- [Cross-Modal UAV Object Tracking: State-Aware Representation Learning and A Unified Benchmark](https://mlsanity.com/paper/2607.18768/) (2026)
- [Template-Search Domain Adaptation via Multi-Stage Feature Alignment for Cross-Modal Object Tracking](https://mlsanity.com/paper/2609.38637/) (2026)

## Abstract (arXiv)

In many visual systems, visual tracking often bases on RGB image sequences, in which some targets are invalid in low-light conditions, and tracking performance is thus affected significantly. Introducing other modalities such as depth and infrared data is an effective way to handle imaging limitations of individual sources, but multi-modal imaging platforms usually require elaborate designs and cannot be applied in many real-world applications at present. Near-infrared (NIR) imaging becomes an essential part of many surveillance cameras, whose imaging is switchable between RGB and NIR based on the light intensity. These two modalities are heterogeneous with very different visual properties and thus bring big challenges for visual tracking. However, existing works have not studied this challenging problem. In this work, we address the cross-modal object tracking problem and contribute a new video dataset, including 654 cross-modal image sequences with over 481K frames in total, and the average video length is more than 735 frames. To promote the research and development of cross-modal object tracking, we propose a new algorithm, which learns the modality-aware target representation to mitigate the appearance gap between RGB and NIR modalities in the tracking process. It is plug-and-play and could thus be flexibly embedded into different tracking frameworks. Extensive experiments on the dataset are conducted, and we demonstrate the effectiveness of the proposed algorithm in two representative tracking frameworks against 17 state-of-the-art tracking methods. We will release the dataset for free academic usage, dataset download link and code will be released soon.