# DeepScale: Online Frame Size Adaptation for Multi-object Tracking on Smart Cameras and Edge Servers

arXiv: https://arxiv.org/abs/2107.10404 · HTML: https://mlsanity.com/paper/2107.10404/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 45 references that are in the map.
- First posted on arXiv 22 Jul 2021.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020)
- [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019)
- [CrowdHuman: A Benchmark for Detecting Human in a Crowd](https://mlsanity.com/paper/1805.00123/) (2018)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [MOTS: Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/1902.03604/) (2019)
- [MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking](https://mlsanity.com/paper/1504.01942/) (2015)
- [Learning to Track with Object Permanence](https://mlsanity.com/paper/2103.14258/) (2021)
- [AR-Net: Adaptive Frame Resolution for Efficient Action Recognition](https://mlsanity.com/paper/2007.15796/) (2020)
- [TubeTK: Adopting Tubes to Track Multi-Object in a One-Step Training Model](https://mlsanity.com/paper/2006.05683/) (2020)
- [DeepCache: Principled Cache for Mobile Deep Vision](https://mlsanity.com/paper/1712.01670/) (2017)
- [Near-Online Multi-target Tracking with Aggregated Local Flow Descriptor](https://mlsanity.com/paper/1504.02340/) (2015)
- [Edge-Host Partitioning of Deep Neural Networks with Feature Space Encoding for Resource-Constrained Internet-of-Things Platforms](https://mlsanity.com/paper/1802.03835/) (2018)
- [AdaScale: Towards Real-time Video Object Detection Using Adaptive Scaling](https://mlsanity.com/paper/1902.02910/) (2019)

## Built on by (most cited first, 4 of 4)

- [Edge Video Analytics: A Survey on Applications, Systems and Enabling Techniques](https://mlsanity.com/paper/2211.15751/) (2022)
- [Optimizing Edge Offloading Decisions for Object Detection](https://mlsanity.com/paper/2410.18919/) (2024)
- [AttTrack: Online Deep Attention Transfer for Multi-object Tracking](https://mlsanity.com/paper/2210.08648/) (2022)
- [Learning Online Policies for Person Tracking in Multi-View Environments](https://mlsanity.com/paper/2312.15858/) (2023)

## Abstract (arXiv)

In surveillance and search and rescue applications, it is important to perform multi-target tracking (MOT) in real-time on low-end devices. Today's MOT solutions employ deep neural networks, which tend to have high computation complexity. Recognizing the effects of frame sizes on tracking performance, we propose DeepScale, a model agnostic frame size selection approach that operates on top of existing fully convolutional network-based trackers to accelerate tracking throughput. In the training stage, we incorporate detectability scores into a one-shot tracker architecture so that DeepScale can learn representation estimations for different frame sizes in a self-supervised manner. During inference, it can adapt frame sizes according to the complexity of visual contents based on user-controlled parameters. To leverage computation resources on edge servers, we propose two computation partition schemes tailored for MOT, namely, edge server only with adaptive frame-size transmission and edge server-assisted tracking. Extensive experiments and benchmark tests on MOT datasets demonstrate the effectiveness and flexibility of DeepScale. Compared to a state-of-the-art tracker, DeepScale++, a variant of DeepScale achieves 1.57X accelerated with only moderate degradation ~2.3 in tracking accuracy on the MOT15 dataset in one configuration. We have implemented and evaluated DeepScale++ and the proposed computation partition schemes on a small-scale testbed consisting of an NVIDIA Jetson TX2 board and a GPU server. The experiments reveal non-trivial trade-offs between tracking performance and latency compared to server-only or smart camera-only solutions.