# Self-Supervised RGB-T Tracking with Cross-Input Consistency

arXiv: https://arxiv.org/abs/2301.11274 · HTML: https://mlsanity.com/paper/2301.11274/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 63 references that are in the map.
- First posted on arXiv 26 Jan 2023.

## Builds on (most cited first, 30 of 31)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [Tracking Emerges by Colorizing Videos](https://mlsanity.com/paper/1806.09594/) (2018)
- [Deeper and Wider Siamese Networks for Real-Time Visual Tracking](https://mlsanity.com/paper/1901.01660/) (2019)
- [SwinTrack: A Simple and Strong Baseline for Transformer Tracking](https://mlsanity.com/paper/2112.00995/) (2021)
- [RGB-T Object Tracking:Benchmark and Baseline](https://mlsanity.com/paper/1805.08982/) (2018)
- [Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline](https://mlsanity.com/paper/2204.04120/) (2022)
- [End-to-end representation learning for Correlation Filter based tracking](https://mlsanity.com/paper/1704.06036/) (2017)
- [Unsupervised Deep Tracking](https://mlsanity.com/paper/1904.01828/) (2019)
- [Multi-Modal Fusion for End-to-End RGB-T Tracking](https://mlsanity.com/paper/1908.11714/) (2019)
- [AutoTrack: Towards High-Performance Visual Tracking for UAV with Automatic Spatio-Temporal Regularization](https://mlsanity.com/paper/2003.12949/) (2020)
- [Challenge-Aware RGBT Tracking](https://mlsanity.com/paper/2007.13143/) (2020)
- [Jointly Modeling Motion and Appearance Cues for Robust RGB-T Tracking](https://mlsanity.com/paper/2007.02041/) (2020)
- [Target-Aware Deep Tracking](https://mlsanity.com/paper/1904.01772/) (2019)
- [Dense Feature Aggregation and Pruning for RGBT Tracking](https://mlsanity.com/paper/1907.10451/) (2019)
- [Multi-Adapter RGBT Tracking](https://mlsanity.com/paper/1907.07485/) (2019)
- [Duality-Gated Mutual Condition Network for RGBT Tracking](https://mlsanity.com/paper/2011.07188/) (2020)
- [RGBT Tracking via Multi-Adapter Network with Hierarchical Divergence Loss](https://mlsanity.com/paper/2011.07189/) (2020)
- [MFGNet: Dynamic Modality-Aware Filter Generation for RGB-T Tracking](https://mlsanity.com/paper/2107.10433/) (2021)
- [Unsupervised Deep Representation Learning for Real-Time Tracking](https://mlsanity.com/paper/2007.11984/) (2020)
- [VIFB: A Visible and Infrared Image Fusion Benchmark](https://mlsanity.com/paper/2002.03322/) (2020)
- [Self-Supervised Multi-Object Tracking with Cross-Input Consistency](https://mlsanity.com/paper/2111.05943/) (2021)
- [Learning to Track Objects from Unlabeled Videos](https://mlsanity.com/paper/2108.12711/) (2021)
- [Learning Deep Multi-Level Similarity for Thermal Infrared Object Tracking](https://mlsanity.com/paper/1906.03568/) (2019)
- [Unsupervised Learning of Accurate Siamese Tracking](https://mlsanity.com/paper/2204.01475/) (2022)
- [DCFNet: Discriminant Correlation Filters Network for Visual Tracking](https://mlsanity.com/paper/1704.04057/) (2017)
- [Self-supervised Object Tracking with Cycle-consistent Siamese Networks](https://mlsanity.com/paper/2008.00637/) (2020)
- [Self-Supervised Tracking via Target-Aware Data Synthesis](https://mlsanity.com/paper/2106.10900/) (2021)

## Abstract (arXiv)

In this paper, we propose a self-supervised RGB-T tracking method. Different from existing deep RGB-T trackers that use a large number of annotated RGB-T image pairs for training, our RGB-T tracker is trained using unlabeled RGB-T video pairs in a self-supervised manner. We propose a novel cross-input consistency-based self-supervised training strategy based on the idea that tracking can be performed using different inputs. Specifically, we construct two distinct inputs using unlabeled RGB-T video pairs. We then track objects using these two inputs to generate results, based on which we construct our cross-input consistency loss. Meanwhile, we propose a reweighting strategy to make our loss function robust to low-quality training samples. We build our tracker on a Siamese correlation filter network. To the best of our knowledge, our tracker is the first self-supervised RGB-T tracker. Extensive experiments on two public RGB-T tracking benchmarks demonstrate that the proposed training strategy is effective. Remarkably, despite training only with a corpus of unlabeled RGB-T video pairs, our tracker outperforms seven supervised RGB-T trackers on the GTOT dataset.