# Detect or Track: Towards Cost-Effective Video Object Detection/Tracking

arXiv: https://arxiv.org/abs/1811.05340 · HTML: https://mlsanity.com/paper/1811.05340/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2018.

## Built on by (most cited first, 18 of 18)

- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [Deep Learning for UAV-based Object Detection and Tracking: A Survey](https://mlsanity.com/paper/2110.12638/) (2021)
- [FOVEA: Foveated Image Magnification for Autonomous Navigation](https://mlsanity.com/paper/2108.12102/) (2021)
- [Real-Time Reasoning Agents in Evolving Environments](https://mlsanity.com/paper/2511.04898/) (2025)
- [SALISA: Saliency-based Input Sampling for Efficient Video Object Detection](https://mlsanity.com/paper/2204.02397/) (2022)
- [QueryProp: Object Query Propagation for High-Performance Video Object Detection](https://mlsanity.com/paper/2207.10959/) (2022)
- [PatchNet -- Short-range Template Matching for Efficient Video Processing](https://mlsanity.com/paper/2103.07371/) (2021)
- [SportsTrack: An Innovative Method for Tracking Athletes in Sports Scenes](https://mlsanity.com/paper/2211.07173/) (2022)
- [Memory Maps for Video Object Detection and Tracking on UAVs](https://mlsanity.com/paper/2303.03508/) (2023)
- [FasterVideo: Efficient Online Joint Object Detection And Tracking](https://mlsanity.com/paper/2204.07394/) (2022)
- [Object Detection Difficulty: Suppressing Over-aggregation for Faster and Better Video Object Detection](https://mlsanity.com/paper/2308.11327/) (2023)
- [Fast and Resource-Efficient Object Tracking on Edge Devices: A Measurement Study](https://mlsanity.com/paper/2309.02666/) (2023)
- [SSGA-Net: Stepwise Spatial Global-local Aggregation Networks for for Autonomous Driving](https://mlsanity.com/paper/2405.18857/) (2024)
- [A Generic Object Re-identification System for Short Videos](https://mlsanity.com/paper/2102.05275/) (2021)
- [VariabilityTrack:Multi-Object Tracking with Variable Speed Object Movement](https://mlsanity.com/paper/2203.06424/) (2022)
- [PLATE: A perception-latency aware estimator,](https://mlsanity.com/paper/2401.13596/) (2024)
- [A comprehensive overview of deep learning models for object detection from videos/images](https://mlsanity.com/paper/2601.14677/) (2026)
- [Albireo: Adaptive, Energy-Efficient Inference Framework for Video Object Detection on the Edge](https://mlsanity.com/paper/2609.29648/) (2026)

## Abstract (arXiv)

State-of-the-art object detectors and trackers are developing fast. Trackers are in general more efficient than detectors but bear the risk of drifting. A question is hence raised -- how to improve the accuracy of video object detection/tracking by utilizing the existing detectors and trackers within a given time budget? A baseline is frame skipping -- detecting every N-th frames and tracking for the frames in between. This baseline, however, is suboptimal since the detection frequency should depend on the tracking quality. To this end, we propose a scheduler network, which determines to detect or track at a certain frame, as a generalization of Siamese trackers. Although being light-weight and simple in structure, the scheduler network is more effective than the frame skipping baselines and flow-based approaches, as validated on ImageNet VID dataset in video object detection/tracking.