# A Reinforcement-Learning-Based Energy-Efficient Framework for Multi-Task Video Analytics Pipeline

arXiv: https://arxiv.org/abs/2104.04443 · HTML: https://mlsanity.com/paper/2104.04443/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 9 Apr 2021.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [Deep Feature Flow for Video Recognition](https://mlsanity.com/paper/1611.07715/) (2016)
- [S3FD: Single Shot Scale-invariant Face Detector](https://mlsanity.com/paper/1708.05237/) (2017)
- [Dynamic Video Segmentation Network](https://mlsanity.com/paper/1804.00931/) (2018)
- [Dynamic Face Video Segmentation via Reinforcement Learning](https://mlsanity.com/paper/1907.01296/) (2019)
- [Shape Constrained Network for Eye Segmentation in the Wild](https://mlsanity.com/paper/1910.05283/) (2019)
- [Face Mask Extraction in Video Sequence](https://mlsanity.com/paper/1807.09207/) (2018)
- [Dilated Convolutions with Lateral Inhibitions for Semantic Image Segmentation](https://mlsanity.com/paper/2006.03708/) (2020)

## Built on by (most cited first, 3 of 3)

- [ILCAS: Imitation Learning-Based Configuration-Adaptive Streaming for Live Video Analytics with Cross-Camera Collaboration](https://mlsanity.com/paper/2308.10068/) (2023)
- [Do Smart Glasses Dream of Sentimental Visions? Deep Emotionship Analysis for Eyewear Devices](https://mlsanity.com/paper/2201.09933/) (2022)
- [MemX: An Attention-Aware Smart Eyewear System for Personalized Moment Auto-capture](https://mlsanity.com/paper/2105.00916/) (2021)

## Abstract (arXiv)

Deep-learning-based video processing has yielded transformative results in recent years. However, the video analytics pipeline is energy-intensive due to high data rates and reliance on complex inference algorithms, which limits its adoption in energy-constrained applications. Motivated by the observation of high and variable spatial redundancy and temporal dynamics in video data streams, we design and evaluate an adaptive-resolution optimization framework to minimize the energy use of multi-task video analytics pipelines. Instead of heuristically tuning the input data resolution of individual tasks, our framework utilizes deep reinforcement learning to dynamically govern the input resolution and computation of the entire video analytics pipeline. By monitoring the impact of varying resolution on the quality of high-dimensional video analytics features, hence the accuracy of video analytics results, the proposed end-to-end optimization framework learns the best non-myopic policy for dynamically controlling the resolution of input video streams to globally optimize energy efficiency. Governed by reinforcement learning, optical flow is incorporated into the framework to minimize unnecessary spatio-temporal redundancy that leads to re-computation, while preserving accuracy. The proposed framework is applied to video instance segmentation which is one of the most challenging computer vision tasks, and achieves better energy efficiency than all baseline methods of similar accuracy on the YouTube-VIS dataset.