# Adaptive Subsampling for ROI-based Visual Tracking: Algorithms and FPGA Implementation

arXiv: https://arxiv.org/abs/2112.09775 · HTML: https://mlsanity.com/paper/2112.09775/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 97 references that are in the map.
- First posted on arXiv 17 Dec 2021.

## Builds on (most cited first, 25 of 25)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [ECO: Efficient Convolution Operators for Tracking](https://mlsanity.com/paper/1611.09224/) (2016)
- [Learning Multi-Domain Convolutional Neural Networks for Visual Tracking](https://mlsanity.com/paper/1510.07945/) (2015)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)
- [Acquisition of Localization Confidence for Accurate Object Detection](https://mlsanity.com/paper/1807.11590/) (2018)
- [Deformable Siamese Attention Networks for Visual Object Tracking](https://mlsanity.com/paper/2004.06711/) (2020)
- [Deep Affinity Network for Multiple Object Tracking](https://mlsanity.com/paper/1810.11780/) (2018)
- [VITAL: VIsual Tracking via Adversarial Learning](https://mlsanity.com/paper/1804.04273/) (2018)
- [μNAS: Constrained Neural Architecture Search for Microcontrollers](https://mlsanity.com/paper/2010.14246/) (2020)
- [Reconfiguring the Imaging Pipeline for Computer Vision](https://mlsanity.com/paper/1705.04352/) (2017)
- [SkyNet: a Hardware-Efficient Method for Object Detection and Tracking on Embedded Systems](https://mlsanity.com/paper/1909.09709/) (2019)
- [A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones](https://mlsanity.com/paper/1805.01831/) (2018)
- [Spatially Supervised Recurrent Convolutional Neural Networks for Visual Object Tracking](https://mlsanity.com/paper/1607.05781/) (2016)
- [SANet: Structure-Aware Network for Visual Tracking](https://mlsanity.com/paper/1611.06878/) (2016)
- [Are FPGAs Suitable for Edge Computing?](https://mlsanity.com/paper/1804.06404/) (2018)
- [Predictive Visual Tracking: A New Benchmark and Baseline Approach](https://mlsanity.com/paper/2103.04508/) (2021)
- [Deep Learning on FPGAs: Past, Present, and Future](https://mlsanity.com/paper/1602.04283/) (2016)
- [A Comparison of CNN-based Face and Head Detectors for Real-Time Video Surveillance Applications](https://mlsanity.com/paper/1809.03336/) (2018)
- [Early Seizure Detection with an Energy-Efficient Convolutional Neural Network on an Implantable Microcontroller](https://mlsanity.com/paper/1806.04549/) (2018)
- [Energy-Efficient Inference on the Edge Exploiting TinyML Capabilities for UAVs](https://mlsanity.com/paper/2111.15481/) (2021)
- [Deep Learning of Appearance Models for Online Object Tracking](https://mlsanity.com/paper/1607.02568/) (2016)

## Abstract (arXiv)

There is tremendous scope for improving the energy efficiency of embedded vision systems by incorporating programmable region-of-interest (ROI) readout in the image sensor design. In this work, we study how ROI programmability can be leveraged for tracking applications by anticipating where the ROI will be located in future frames and switching pixels off outside of this region. We refer to this process of ROI prediction and corresponding sensor configuration as adaptive subsampling. Our adaptive subsampling algorithms comprise an object detector and an ROI predictor (Kalman filter) which operate in conjunction to optimize the energy efficiency of the vision pipeline with the end task being object tracking. To further facilitate the implementation of our adaptive algorithms in real life, we select a candidate algorithm and map it onto an FPGA. Leveraging Xilinx Vitis AI tools, we designed and accelerated a YOLO object detector-based adaptive subsampling algorithm. In order to further improve the algorithm post-deployment, we evaluated several competing baselines on the OTB100 and LaSOT datasets. We found that coupling the ECO tracker with the Kalman filter has a competitive AUC score of 0.4568 and 0.3471 on the OTB100 and LaSOT datasets respectively. Further, the power efficiency of this algorithm is on par with, and in a couple of instances superior to, the other baselines. The ECO-based algorithm incurs a power consumption of approximately 4 W averaged across both datasets while the YOLO-based approach requires power consumption of approximately 6 W (as per our power consumption model). In terms of accuracy-latency tradeoff, the ECO-based algorithm provides near-real-time performance (19.23 FPS) while managing to attain competitive tracking precision.