# Unsupervised Deep Representation Learning for Real-Time Tracking

arXiv: https://arxiv.org/abs/2007.11984 · HTML: https://mlsanity.com/paper/2007.11984/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2020.

## Built on by (most cited first, 29 of 29)

- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples](https://mlsanity.com/paper/2103.05905/) (2021)
- [ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows](https://mlsanity.com/paper/2103.16877/) (2021)
- [Disentangled Cycle Consistency for Highly-realistic Virtual Try-On](https://mlsanity.com/paper/2103.09479/) (2021)
- [Do Different Tracking Tasks Require Different Appearance Models?](https://mlsanity.com/paper/2107.02156/) (2021)
- [Unsupervised Domain Adaptation for Nighttime Aerial Tracking](https://mlsanity.com/paper/2203.10541/) (2022)
- [Visual Object Tracking with Discriminative Filters and Siamese Networks: A Survey and Outlook](https://mlsanity.com/paper/2112.02838/) (2021)
- [Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2110.05340/) (2021)
- [IoU Attack: Towards Temporally Coherent Black-Box Adversarial Attack for Visual Object Tracking](https://mlsanity.com/paper/2103.14938/) (2021)
- [Diff-Tracker: Text-to-Image Diffusion Models are Unsupervised Trackers](https://mlsanity.com/paper/2407.08394/) (2024)
- [Learning to Track Objects from Unlabeled Videos](https://mlsanity.com/paper/2108.12711/) (2021)
- [Unsupervised Learning of Accurate Siamese Tracking](https://mlsanity.com/paper/2204.01475/) (2022)
- [SurgT challenge: Benchmark of Soft-Tissue Trackers for Robotic Surgery](https://mlsanity.com/paper/2302.03022/) (2023)
- [SiamMask: A Framework for Fast Online Object Tracking and Segmentation](https://mlsanity.com/paper/2207.02088/) (2022)
- [Predicting Video Slot Attention Queries from Random Slot-Feature Pairs](https://mlsanity.com/paper/2508.01345/) (2025)
- [Securing the Skies: A Comprehensive Survey on Anti-UAV Methods, Benchmarking, and Future Directions](https://mlsanity.com/paper/2504.11967/) (2025)
- [Self-Supervised Tracking via Target-Aware Data Synthesis](https://mlsanity.com/paper/2106.10900/) (2021)
- [An Informative Tracking Benchmark](https://mlsanity.com/paper/2112.06467/) (2021)
- [GUSOT: Green and Unsupervised Single Object Tracking for Long Video Sequences](https://mlsanity.com/paper/2207.07629/) (2022)
- [Unsupervised Lightweight Single Object Tracking with UHP-SOT++](https://mlsanity.com/paper/2111.07548/) (2021)
- [MixCycle: Mixup Assisted Semi-Supervised 3D Single Object Tracking with Cycle Consistency](https://mlsanity.com/paper/2303.09219/) (2023)
- [Exploiting Lightweight Hierarchical ViT and Dynamic Framework for Efficient Visual Tracking](https://mlsanity.com/paper/2506.20381/) (2025)
- [Boosting Self-Supervised Tracking with Contextual Prompts and Noise Learning](https://mlsanity.com/paper/2605.06092/) (2026)
- [Learning to Track Instance from Single Nature Language Description](https://mlsanity.com/paper/2605.07064/) (2026)
- [Leveraging Text-to-Image Diffusion Models for Unsupervised Visual Object Tracking](https://mlsanity.com/paper/2605.26933/) (2026)
- [Person Monitoring by Full Body Tracking in Uniform Crowd Environment](https://mlsanity.com/paper/2209.01274/) (2022)
- [Self-Supervised RGB-T Tracking with Cross-Input Consistency](https://mlsanity.com/paper/2301.11274/) (2023)
- [Unsupervised Green Object Tracker (GOT) without Offline Pre-training](https://mlsanity.com/paper/2309.09078/) (2023)
- [Learning Tracking Representations from Single Point Annotations](https://mlsanity.com/paper/2404.09504/) (2024)

## Abstract (arXiv)

The advancement of visual tracking has continuously been brought by deep learning models. Typically, supervised learning is employed to train these models with expensive labeled data. In order to reduce the workload of manual annotations and learn to track arbitrary objects, we propose an unsupervised learning method for visual tracking. The motivation of our unsupervised learning is that a robust tracker should be effective in bidirectional tracking. Specifically, the tracker is able to forward localize a target object in successive frames and backtrace to its initial position in the first frame. Based on such a motivation, in the training process, we measure the consistency between forward and backward trajectories to learn a robust tracker from scratch merely using unlabeled videos. We build our framework on a Siamese correlation filter network, and propose a multi-frame validation scheme and a cost-sensitive loss to facilitate unsupervised learning. Without bells and whistles, the proposed unsupervised tracker achieves the baseline accuracy as classic fully supervised trackers while achieving a real-time speed. Furthermore, our unsupervised framework exhibits a potential in leveraging more unlabeled or weakly labeled data to further improve the tracking accuracy.