# Deep Reinforcement Learning for Visual Object Tracking in Videos

arXiv: https://arxiv.org/abs/1701.08936 · HTML: https://mlsanity.com/paper/1701.08936/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2017.

## Built on by (most cited first, 8 of 8)

- [DIMBA: Discretely Masked Black-Box Attack in Single Object Tracking](https://mlsanity.com/paper/2207.08044/) (2022)
- [AlignSAM: Aligning Segment Anything Model to Open Context via Reinforcement Learning](https://mlsanity.com/paper/2406.00480/) (2024)
- [Beyond Greedy Search: Tracking by Multi-Agent Reinforcement Learning-based Beam Search](https://mlsanity.com/paper/2205.09676/) (2022)
- [Open-World Drone Active Tracking with Goal-Centered Rewards](https://mlsanity.com/paper/2412.00744/) (2024)
- [RAP: Runtime Adaptive Pruning for LLM Inference](https://mlsanity.com/paper/2505.17138/) (2025)
- [IO Transformer: Evaluating SwinV2-Based Reward Models for Computer Vision](https://mlsanity.com/paper/2411.00252/) (2024)
- [Object Tracking in a 360o View: A Novel Perspective on Bridging the Gap to Biomedical Advancements](https://mlsanity.com/paper/2412.01119/) (2024)
- [GoStop: Reinforcement Learning for Adaptive Temporal Aggregation in Event-Based Feature Tracking](https://mlsanity.com/paper/2607.15699/) (2026)

## Abstract (arXiv)

In this paper we introduce a fully end-to-end approach for visual tracking in videos that learns to predict the bounding box locations of a target object at every frame. An important insight is that the tracking problem can be considered as a sequential decision-making process and historical semantics encode highly relevant information for future decisions. Based on this intuition, we formulate our model as a recurrent convolutional neural network agent that interacts with a video overtime, and our model can be trained with reinforcement learning (RL) algorithms to learn good tracking policies that pay attention to continuous, inter-frame correlation and maximize tracking performance in the long run. The proposed tracking algorithm achieves state-of-the-art performance in an existing tracking benchmark and operates at frame-rates faster than real-time. To the best of our knowledge, our tracker is the first neural-network tracker that combines convolutional and recurrent networks with RL algorithms.