# FrameHopper: Selective Processing of Video Frames in Detection-driven Real-Time Video Analytics

arXiv: https://arxiv.org/abs/2203.11493 · HTML: https://mlsanity.com/paper/2203.11493/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Self-trained Deep Ordinal Regression for End-to-End Video Anomaly Detection](https://mlsanity.com/paper/2003.06780/) (2020)
- [NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision](https://mlsanity.com/paper/1810.10090/) (2018)
- [Appearance-Preserving 3D Convolution for Video-based Person Re-identification](https://mlsanity.com/paper/2007.08434/) (2020)
- [Temporal Complementary Learning for Video Person Re-Identification](https://mlsanity.com/paper/2007.09357/) (2020)
- [NoScope: Optimizing Neural Network Queries over Video at Scale](https://mlsanity.com/paper/1703.02529/) (2017)
- [Scaling Video Analytics on Constrained Edge Nodes](https://mlsanity.com/paper/1905.13536/) (2019)
- [Efficient Video Classification Using Fewer Frames](https://mlsanity.com/paper/1902.10640/) (2019)
- [Combining Compositional Models and Deep Networks For Robust Object Classification under Occlusion](https://mlsanity.com/paper/1905.11826/) (2019)
- [RefineFace: Refinement Neural Network for High Performance Face Detection](https://mlsanity.com/paper/1909.04376/) (2019)
- [Learning Complexity-Aware Cascades for Deep Pedestrian Detection](https://mlsanity.com/paper/1507.05348/) (2015)
- [FFNet: Video Fast-Forwarding via Reinforcement Learning](https://mlsanity.com/paper/1805.02792/) (2018)
- [Rare Event Detection using Disentangled Representation Learning](https://mlsanity.com/paper/1812.01285/) (2018)
- [Traffic-Aware Multi-Camera Tracking of Vehicles Based on ReID and Camera Link Model](https://mlsanity.com/paper/2008.09785/) (2020)
- [NTIRE 2021 Multi-modal Aerial View Object Classification Challenge](https://mlsanity.com/paper/2107.01189/) (2021)

## Built on by (most cited first, 4 of 4)

- [Context-aware Multi-Model Object Detection for Diversely Heterogeneous Compute Systems](https://mlsanity.com/paper/2402.07415/) (2024)
- [Fast and Resource-Efficient Object Tracking on Edge Devices: A Measurement Study](https://mlsanity.com/paper/2309.02666/) (2023)
- [Efficient License Plate Recognition in Videos Using Visual Rhythm and Accumulative Line Analysis](https://mlsanity.com/paper/2501.04750/) (2025)
- [Albireo: Adaptive, Energy-Efficient Inference Framework for Video Object Detection on the Edge](https://mlsanity.com/paper/2609.29648/) (2026)

## Abstract (arXiv)

Detection-driven real-time video analytics require continuous detection of objects contained in the video frames using deep learning models like YOLOV3, EfficientDet. However, running these detectors on each and every frame in resource-constrained edge devices is computationally intensive. By taking the temporal correlation between consecutive video frames into account, we note that detection outputs tend to be overlapping in successive frames. Elimination of similar consecutive frames will lead to a negligible drop in performance while offering significant performance benefits by reducing overall computation and communication costs. The key technical questions are, therefore, (a) how to identify which frames to be processed by the object detector, and (b) how many successive frames can be skipped (called skip-length) once a frame is selected to be processed. The overall goal of the process is to keep the error due to skipping frames as small as possible. We introduce a novel error vs processing rate optimization problem with respect to the object detection task that balances between the error rate and the fraction of frames filtering. Subsequently, we propose an off-line Reinforcement Learning (RL)-based algorithm to determine these skip-lengths as a state-action policy of the RL agent from a recorded video and then deploy the agent online for live video streams. To this end, we develop FrameHopper, an edge-cloud collaborative video analytics framework, that runs a lightweight trained RL agent on the camera and passes filtered frames to the server where the object detection model runs for a set of applications. We have tested our approach on a number of live videos captured from real-life scenarios and show that FrameHopper processes only a handful of frames but produces detection results closer to the oracle solution and outperforms recent state-of-the-art solutions in most cases.