# Tracking as Online Decision-Making: Learning a Policy from Streaming Videos with Reinforcement Learning

arXiv: https://arxiv.org/abs/1707.04991 · HTML: https://mlsanity.com/paper/1707.04991/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2017.

## Built on by (most cited first, 14 of 14)

- [Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2406.08085/) (2024)
- [Imitation Learning from Imperfect Demonstration](https://mlsanity.com/paper/1901.09387/) (2019)
- [Flash-VStream: Efficient Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2506.23825/) (2025)
- [Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning](https://mlsanity.com/paper/2110.01770/) (2021)
- [Video Summarization through Reinforcement Learning with a 3D Spatio-Temporal U-Net](https://mlsanity.com/paper/2106.10528/) (2021)
- [PFRL: Pose-Free Reinforcement Learning for 6D Pose Estimation](https://mlsanity.com/paper/2102.12096/) (2021)
- [Towards Sequence-Level Training for Visual Tracking](https://mlsanity.com/paper/2208.05810/) (2022)
- [Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networks](https://mlsanity.com/paper/2206.02887/) (2022)
- [Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulation](https://mlsanity.com/paper/2402.07127/) (2024)
- [Edge-Based Video Analytics: A Survey](https://mlsanity.com/paper/2303.14329/) (2023)
- [Beyond Greedy Search: Tracking by Multi-Agent Reinforcement Learning-based Beam Search](https://mlsanity.com/paper/2205.09676/) (2022)
- [A Heuristically Assisted Deep Reinforcement Learning Approach for Network Slice Placement](https://mlsanity.com/paper/2105.06741/) (2021)
- [Benchmarking Deep Trackers on Aerial Videos](https://mlsanity.com/paper/2103.12924/) (2021)
- [Decouple and Cache: KV Cache Construction for Streaming Video Understanding](https://mlsanity.com/paper/2605.01858/) (2026)

## Abstract (arXiv)

We formulate tracking as an online decision-making process, where a tracking agent must follow an object despite ambiguous image frames and a limited computational budget. Crucially, the agent must decide where to look in the upcoming frames, when to reinitialize because it believes the target has been lost, and when to update its appearance model for the tracked object. Such decisions are typically made heuristically. Instead, we propose to learn an optimal decision-making policy by formulating tracking as a partially observable decision-making process (POMDP). We learn policies with deep reinforcement learning algorithms that need supervision (a reward signal) only when the track has gone awry. We demonstrate that sparse rewards allow us to quickly train on massive datasets, several orders of magnitude more than past work. Interestingly, by treating the data source of Internet videos as unlimited streams, we both learn and evaluate our trackers in a single, unified computational stream.