# Learning Progressive Adaptation for Multi-Modal Tracking

arXiv: https://arxiv.org/abs/2603.21100 · HTML: https://mlsanity.com/paper/2603.21100/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 40 of its 67 references that are in the map.
- First posted on arXiv 22 Mar 2026.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [Probabilistic Regression for Visual Tracking](https://mlsanity.com/paper/2003.12565/) (2020)
- [Visual Prompt Multi-Modal Tracking](https://mlsanity.com/paper/2303.10826/) (2023)
- [VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows](https://mlsanity.com/paper/2108.05015/) (2021)
- [ODTrack: Online Dense Temporal Token Learning for Visual Tracking](https://mlsanity.com/paper/2401.01686/) (2024)
- [RGB-T Object Tracking:Benchmark and Baseline](https://mlsanity.com/paper/1805.08982/) (2018)
- [Inception Transformer](https://mlsanity.com/paper/2205.12956/) (2022)
- [LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking](https://mlsanity.com/paper/2104.13202/) (2021)
- [Object Tracking by Jointly Exploiting Frame and Event Domain](https://mlsanity.com/paper/2109.09052/) (2021)
- [Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline](https://mlsanity.com/paper/2309.14611/) (2023)
- [Recurrent Relational Networks](https://mlsanity.com/paper/1711.08028/) (2017)
- [RGBT Salient Object Detection: A Large-scale Dataset and Benchmark](https://mlsanity.com/paper/2007.03262/) (2020)
- [DepthTrack : Unveiling the Power of RGBD Tracking](https://mlsanity.com/paper/2108.13962/) (2021)
- [Relational recurrent neural networks](https://mlsanity.com/paper/1806.01822/) (2018)
- [OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning](https://mlsanity.com/paper/2403.09634/) (2024)
- [Prompting for Multi-Modal Tracking](https://mlsanity.com/paper/2207.14571/) (2022)
- [Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking](https://mlsanity.com/paper/2412.15691/) (2024)
- [Bi-directional Adapter for Multi-modal Tracking](https://mlsanity.com/paper/2312.10611/) (2023)
- [SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking](https://mlsanity.com/paper/2403.16002/) (2024)
- [RGBD1K: A Large-scale Dataset and Benchmark for RGB-D Object Tracking](https://mlsanity.com/paper/2208.09787/) (2022)
- [Prototypical Contrast Adaptation for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2207.06654/) (2022)
- [Seeing Glass: Joint Point Cloud and Depth Completion for Transparent Objects](https://mlsanity.com/paper/2110.00087/) (2021)

## Abstract (arXiv)

Due to the limited availability of paired multi-modal data, multi-modal trackers are typically built by adopting pre-trained RGB models with parameter-efficient fine-tuning modules. However, these fine-tuning methods overlook advanced adaptations for applying RGB pre-trained models and fail to modulate a single specific modality, cross-modal interactions, and the prediction head. To address the issues, we propose to perform Progressive Adaptation for Multi-Modal Tracking (PATrack). This innovative approach incorporates modality-dependent, modality-entangled, and task-level adapters, effectively bridging the gap in adapting RGB pre-trained networks to multi-modal data through a progressive strategy. Specifically, modality-specific information is enhanced through the modality-dependent adapter, decomposing the high- and low-frequency components, which ensures a more robust feature representation within each modality. The inter-modal interactions are introduced in the modality-entangled adapter, which implements a cross-attention operation guided by inter-modal shared information, ensuring the reliability of features conveyed between modalities. Additionally, recognising that the strong inductive bias of the prediction head does not adapt to the fused information, a task-level adapter specific to the prediction head is introduced. In summary, our design integrates intra-modal, inter-modal, and task-level adapters into a unified framework. Extensive experiments on RGB+Thermal, RGB+Depth, and RGB+Event tracking tasks demonstrate that our method shows impressive performance against state-of-the-art methods. Code is available at https://github.com/ouha1998/Learning-Progressive-Adaptation-for-Multi-Modal-Tracking.