# Cross-Modal Object Tracking via Modality-Aware Fusion Network and A Large-Scale Dataset

arXiv: https://arxiv.org/abs/2312.14446 · HTML: https://mlsanity.com/paper/2312.14446/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 73 references that are in the map.
- First posted on arXiv 22 Dec 2023.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [Ocean: Object-aware Anchor-free Tracking](https://mlsanity.com/paper/2006.10721/) (2020)
- [Learning Multi-Domain Convolutional Neural Networks for Visual Tracking](https://mlsanity.com/paper/1510.07945/) (2015)
- [Siamese Box Adaptive Network for Visual Tracking](https://mlsanity.com/paper/2003.06761/) (2020)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)
- [Acquisition of Localization Confidence for Accurate Object Detection](https://mlsanity.com/paper/1807.11590/) (2018)
- [Deeper and Wider Siamese Networks for Real-Time Visual Tracking](https://mlsanity.com/paper/1901.01660/) (2019)
- [AiATrack: Attention in Attention for Transformer Visual Tracking](https://mlsanity.com/paper/2207.09603/) (2022)
- [VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows](https://mlsanity.com/paper/2108.05015/) (2021)
- [Return of the Devil in the Details: Delving Deep into Convolutional Nets](https://mlsanity.com/paper/1405.3531/) (2014)
- [Transforming Model Prediction for Tracking](https://mlsanity.com/paper/2203.11192/) (2022)
- [LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking](https://mlsanity.com/paper/2104.13202/) (2021)
- [Generalized Relation Modeling for Transformer Tracking](https://mlsanity.com/paper/2303.16580/) (2023)
- [Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline](https://mlsanity.com/paper/2204.04120/) (2022)
- [Robust Object Modeling for Visual Tracking](https://mlsanity.com/paper/2308.05140/) (2023)

## Built on by (most cited first, 8 of 8)

- [VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusion](https://mlsanity.com/paper/2502.18042/) (2025)
- [Omni Survey for Multimodality Analysis in Visual Object Tracking](https://mlsanity.com/paper/2508.13000/) (2025)
- [OmniScene: Attention-Augmented Multimodal 4D Scene Understanding for Autonomous Driving](https://mlsanity.com/paper/2509.19973/) (2025)
- [SwiTrack: Tri-State Switch for Cross-Modal Object Tracking](https://mlsanity.com/paper/2511.16227/) (2025)
- [UAV traffic scene understanding: A regulation embedded multi-modal network and a unified benchmark](https://mlsanity.com/paper/2603.10722/) (2026)
- [Unified Multimodal Visual Tracking with Dual Mixture-of-Experts](https://mlsanity.com/paper/2605.03716/) (2026)
- [Cross-Modal UAV Object Tracking: State-Aware Representation Learning and A Unified Benchmark](https://mlsanity.com/paper/2607.18768/) (2026)
- [Template-Search Domain Adaptation via Multi-Stage Feature Alignment for Cross-Modal Object Tracking](https://mlsanity.com/paper/2609.38637/) (2026)

## Abstract (arXiv)

Visual tracking often faces challenges such as invalid targets and decreased performance in low-light conditions when relying solely on RGB image sequences. While incorporating additional modalities like depth and infrared data has proven effective, existing multi-modal imaging platforms are complex and lack real-world applicability. In contrast, near-infrared (NIR) imaging, commonly used in surveillance cameras, can switch between RGB and NIR based on light intensity. However, tracking objects across these heterogeneous modalities poses significant challenges, particularly due to the absence of modality switch signals during tracking. To address these challenges, we propose an adaptive cross-modal object tracking algorithm called Modality-Aware Fusion Network (MAFNet). MAFNet efficiently integrates information from both RGB and NIR modalities using an adaptive weighting mechanism, effectively bridging the appearance gap and enabling a modality-aware target representation. It consists of two key components: an adaptive weighting module and a modality-specific representation module......