# Cross-modulated Attention Transformer for RGBT Tracking

arXiv: https://arxiv.org/abs/2408.02222 · HTML: https://mlsanity.com/paper/2408.02222/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 48 references that are in the map.
- First posted on arXiv 5 Aug 2024.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022)
- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [Visual Prompt Multi-Modal Tracking](https://mlsanity.com/paper/2303.10826/) (2023)
- [AiATrack: Attention in Attention for Transformer Visual Tracking](https://mlsanity.com/paper/2207.09603/) (2022)
- [RGB-T Object Tracking:Benchmark and Baseline](https://mlsanity.com/paper/1805.08982/) (2018)
- [LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking](https://mlsanity.com/paper/2104.13202/) (2021)
- [Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline](https://mlsanity.com/paper/2204.04120/) (2022)
- [LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Search](https://mlsanity.com/paper/2104.14545/) (2021)
- [Single-Model and Any-Modality for Video Object Tracking](https://mlsanity.com/paper/2311.15851/) (2023)
- [OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning](https://mlsanity.com/paper/2403.09634/) (2024)
- [Multi-Modal Fusion for End-to-End RGB-T Tracking](https://mlsanity.com/paper/1908.11714/) (2019)
- [MixFormerV2: Efficient Fully Transformer Tracking](https://mlsanity.com/paper/2305.15896/) (2023)
- [Challenge-Aware RGBT Tracking](https://mlsanity.com/paper/2007.13143/) (2020)
- [Compact Transformer Tracker with Correlative Masked Modeling](https://mlsanity.com/paper/2301.10938/) (2023)
- [SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking](https://mlsanity.com/paper/2403.16002/) (2024)
- [Dense Feature Aggregation and Pruning for RGBT Tracking](https://mlsanity.com/paper/1907.10451/) (2019)
- [Multi-Adapter RGBT Tracking](https://mlsanity.com/paper/1907.07485/) (2019)
- [SparseTT: Visual Tracking with Sparse Transformers](https://mlsanity.com/paper/2205.03776/) (2022)
- [Duality-Gated Mutual Condition Network for RGBT Tracking](https://mlsanity.com/paper/2011.07188/) (2020)
- [RGBT Tracking via Multi-Adapter Network with Hierarchical Divergence Loss](https://mlsanity.com/paper/2011.07189/) (2020)

## Built on by (most cited first, 14 of 14)

- [Adaptive Perception for Unified Visual Multi-modal Object Tracking](https://mlsanity.com/paper/2502.06583/) (2025)
- [CADTrack: Learning Contextual Aggregation with Deformable Alignment for Robust RGBT Tracking](https://mlsanity.com/paper/2511.17967/) (2025)
- [SEATrack: Simple, Efficient, and Adaptive Multimodal Tracker](https://mlsanity.com/paper/2604.12502/) (2026)
- [Omni Survey for Multimodality Analysis in Visual Object Tracking](https://mlsanity.com/paper/2508.13000/) (2025)
- [BTMTrack: Robust RGB-T Tracking via Dual-template Bridging and Temporal-Modal Candidate Elimination](https://mlsanity.com/paper/2501.03616/) (2025)
- [Learning Frequency and Memory-Aware Prompts for Multi-Modal Object Tracking](https://mlsanity.com/paper/2506.23972/) (2025)
- [DMTrack: Spatio-Temporal Multimodal Tracking via Dual-Adapter](https://mlsanity.com/paper/2508.01592/) (2025)
- [Tracking and Segmenting Anything in Any Modality](https://mlsanity.com/paper/2511.19475/) (2025)
- [Spatio-Temporal Conditional Denoising Transformer for Modality-Missing RGBT Tracking](https://mlsanity.com/paper/2607.24701/) (2026)
- [Lightweight RGB-T Tracking with Mobile Vision Transformers](https://mlsanity.com/paper/2506.19154/) (2025)
- [RT-RMOT: A Dataset and Framework for RGB-Thermal Referring Multi-Object Tracking](https://mlsanity.com/paper/2602.22033/) (2026)
- [Sparse-Dense Mixture of Experts Adapter for Multi-Modal Tracking](https://mlsanity.com/paper/2603.13719/) (2026)
- [DRGBT-1K: A Large-scale High-quality Benchmark for Dynamic RGBT Tracking](https://mlsanity.com/paper/2607.19772/) (2026)
- [End-to-End Self-Supervised RGB-T Tracking without Modality Misleading](https://mlsanity.com/paper/2609.37162/) (2026)

## Abstract (arXiv)

Existing Transformer-based RGBT trackers achieve remarkable performance benefits by leveraging self-attention to extract uni-modal features and cross-attention to enhance multi-modal feature interaction and template-search correlation computation. Nevertheless, the independent search-template correlation calculations ignore the consistency between branches, which can result in ambiguous and inappropriate correlation weights. It not only limits the intra-modal feature representation, but also harms the robustness of cross-attention for multi-modal feature interaction and search-template correlation computation. To address these issues, we propose a novel approach called Cross-modulated Attention Transformer (CAFormer), which performs intra-modality self-correlation, inter-modality feature interaction, and search-template correlation computation in a unified attention model, for RGBT tracking. In particular, we first independently generate correlation maps for each modality and feed them into the designed Correlation Modulated Enhancement module, modulating inaccurate correlation weights by seeking the consensus between modalities. Such kind of design unifies self-attention and cross-attention schemes, which not only alleviates inaccurate attention weight computation in self-attention but also eliminates redundant computation introduced by extra cross-attention scheme. In addition, we propose a collaborative token elimination strategy to further improve tracking inference efficiency and accuracy. Extensive experiments on five public RGBT tracking benchmarks show the outstanding performance of the proposed CAFormer against state-of-the-art methods.