# Learning Policies for Adaptive Tracking with Deep Feature Cascades

arXiv: https://arxiv.org/abs/1708.02973 · HTML: https://mlsanity.com/paper/1708.02973/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Aug 2017.

## Built on by (most cited first, 19 of 19)

- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [Weakly Supervised Object Localization and Detection: A Survey](https://mlsanity.com/paper/2104.07918/) (2021)
- [Visual Object Tracking with Discriminative Filters and Siamese Networks: A Survey and Outlook](https://mlsanity.com/paper/2112.02838/) (2021)
- [A Finite Sample Complexity Bound for Distributionally Robust Q-learning](https://mlsanity.com/paper/2302.13203/) (2023)
- [FOVEA: Foveated Image Magnification for Autonomous Navigation](https://mlsanity.com/paper/2108.12102/) (2021)
- [Derivative-Free Reinforcement Learning: A Review](https://mlsanity.com/paper/2102.05710/) (2021)
- [Towards Sequence-Level Training for Visual Tracking](https://mlsanity.com/paper/2208.05810/) (2022)
- [Siamese Object Tracking for Unmanned Aerial Vehicle: A Review and Comprehensive Analysis](https://mlsanity.com/paper/2205.04281/) (2022)
- [Exploring Dynamic Transformer for Efficient Object Tracking](https://mlsanity.com/paper/2403.17651/) (2024)
- [Adaptive Siamese Tracking with a Compact Latent Network](https://mlsanity.com/paper/2302.00930/) (2023)
- [Distilling Channels for Efficient Deep Tracking](https://mlsanity.com/paper/2409.11785/) (2024)
- [Learning Task-preferred Inference Routes for Gradient De-conflict in Multi-output DNNs](https://mlsanity.com/paper/2305.19844/) (2023)
- [Exploiting Lightweight Hierarchical ViT and Dynamic Framework for Efficient Visual Tracking](https://mlsanity.com/paper/2506.20381/) (2025)
- [RELO: Reinforcement Learning to Localize for Visual Object Tracking](https://mlsanity.com/paper/2605.07379/) (2026)
- [Sparsely ensembled convolutional neural network classifiers via reinforcement learning](https://mlsanity.com/paper/2102.03921/) (2021)
- [Benchmarking Deep Trackers on Aerial Videos](https://mlsanity.com/paper/2103.12924/) (2021)
- [Single Object Tracking Research: A Survey](https://mlsanity.com/paper/2204.11410/) (2022)
- [Adaptive Remote Sensing Image Attribute Learning for Active Object Detection](https://mlsanity.com/paper/2101.06438/) (2021)
- [β-Multivariational Autoencoder for Entangled Representation Learning in Video Frames](https://mlsanity.com/paper/2211.12627/) (2022)

## Abstract (arXiv)

Visual object tracking is a fundamental and time-critical vision task. Recent years have seen many shallow tracking methods based on real-time pixel-based correlation filters, as well as deep methods that have top performance but need a high-end GPU. In this paper, we learn to improve the speed of deep trackers without losing accuracy. Our fundamental insight is to take an adaptive approach, where easy frames are processed with cheap features (such as pixel values), while challenging frames are processed with invariant but expensive deep features. We formulate the adaptive tracking problem as a decision-making process, and learn an agent to decide whether to locate objects with high confidence on an early layer, or continue processing subsequent layers of a network. This significantly reduces the feed-forward cost for easy frames with distinct or slow-moving objects. We train the agent offline in a reinforcement learning fashion, and further demonstrate that learning all deep layers (so as to provide good features for adaptive tracking) can lead to near real-time average tracking speed of 23 fps on a single CPU while achieving state-of-the-art performance. Perhaps most tellingly, our approach provides a 100X speedup for almost 50% of the time, indicating the power of an adaptive approach.