# Context Matters: Refining Object Detection in Video with Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1607.04648 · HTML: https://mlsanity.com/paper/1607.04648/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2016.

## Built on by (most cited first, 8 of 8)

- [Deep Learning for UAV-based Object Detection and Tracking: A Survey](https://mlsanity.com/paper/2110.12638/) (2021)
- [A Guide to Image and Video based Small Object Detection using Deep Learning : Case Study of Maritime Surveillance](https://mlsanity.com/paper/2207.12926/) (2022)
- [REX: Revisiting Budgeted Training with an Improved Schedule](https://mlsanity.com/paper/2107.04197/) (2021)
- [Objects do not disappear: Video object detection by single-frame object location anticipation](https://mlsanity.com/paper/2308.04770/) (2023)
- [Few-shot Online Anomaly Detection and Segmentation](https://mlsanity.com/paper/2403.18201/) (2024)
- [Temp-Frustum Net: 3D Object Detection with Temporal Fusion](https://mlsanity.com/paper/2104.12106/) (2021)
- [Instant Video Models: Universal Adapters for Stabilizing Image-Based Networks](https://mlsanity.com/paper/2512.03014/) (2025)
- [FIVE-VLA: Fast and EffectIVE Autonomous Driving with Recurrent Action Memory](https://mlsanity.com/paper/2609.18623/) (2026)

## Abstract (arXiv)

Given the vast amounts of video available online, and recent breakthroughs in object detection with static images, object detection in video offers a promising new frontier. However, motion blur and compression artifacts cause substantial frame-level variability, even in videos that appear smooth to the eye. Additionally, video datasets tend to have sparsely annotated frames. We present a new framework for improving object detection in videos that captures temporal context and encourages consistency of predictions. First, we train a pseudo-labeler, that is, a domain-adapted convolutional neural network for object detection. The pseudo-labeler is first trained individually on the subset of labeled frames, and then subsequently applied to all frames. Then we train a recurrent neural network that takes as input sequences of pseudo-labeled frames and optimizes an objective that encourages both accuracy on the target frame and consistency across consecutive frames. The approach incorporates strong supervision of target frames, weak-supervision on context frames, and regularization via a smoothness penalty. Our approach achieves mean Average Precision (mAP) of 68.73, an improvement of 7.1 over the strongest image-based baselines for the Youtube-Video Objects dataset. Our experiments demonstrate that neighboring frames can provide valuable information, even absent labels.