# Adaptive Scheduling for Edge-Assisted DNN Serving

arXiv: https://arxiv.org/abs/2304.09961 · HTML: https://mlsanity.com/paper/2304.09961/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 75 references that are in the map.
- First posted on arXiv 19 Apr 2023.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [Quantized Convolutional Neural Networks for Mobile Devices](https://mlsanity.com/paper/1512.06473/) (2015)
- [Low-bit Quantization of Neural Networks for Efficient Inference](https://mlsanity.com/paper/1902.06822/) (2019)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation](https://mlsanity.com/paper/2004.09602/) (2020)
- [Towards Accurate Multi-person Pose Estimation in the Wild](https://mlsanity.com/paper/1701.01779/) (2017)
- [Human Pose Estimation with Iterative Error Feedback](https://mlsanity.com/paper/1507.06550/) (2015)
- [Serving DNNs like Clockwork: Performance Predictability from the Bottom Up](https://mlsanity.com/paper/2006.02464/) (2020)
- [NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision](https://mlsanity.com/paper/1810.10090/) (2018)
- [DeepCache: Principled Cache for Mobile Deep Vision](https://mlsanity.com/paper/1712.01670/) (2017)
- [SDCNet: Video Prediction Using Spatially-Displaced Convolution](https://mlsanity.com/paper/1811.00684/) (2018)
- [Efficient Uncertainty Estimation for Semantic Segmentation in Videos](https://mlsanity.com/paper/1807.11037/) (2018)
- [NoScope: Optimizing Neural Network Queries over Video at Scale](https://mlsanity.com/paper/1703.02529/) (2017)
- [LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inference](https://mlsanity.com/paper/2010.13103/) (2020)
- [Latency and Throughput Characterization of Convolutional Neural Networks for Mobile Computer Vision](https://mlsanity.com/paper/1803.09492/) (2018)

## Abstract (arXiv)

Deep neural networks (DNNs) have been widely used in various video analytic tasks. These tasks demand real-time responses. Due to the limited processing power on mobile devices, a common way to support such real-time analytics is to offload the processing to an edge server. This paper examines how to speed up the edge server DNN processing for multiple clients. In particular, we observe batching multiple DNN requests significantly speeds up the processing time. Based on this observation, we first design a novel scheduling algorithm to exploit the batching benefits of all requests that run the same DNN. This is compelling since there are only a handful of DNNs and many requests tend to use the same DNN. Our algorithms are general and can support different objectives, such as minimizing the completion time or maximizing the on-time ratio. We then extend our algorithm to handle requests that use different DNNs with or without shared layers. Finally, we develop a collaborative approach to further improve performance by adaptively processing some of the requests or portions of the requests locally at the clients. This is especially useful when the network and/or server is congested. Our implementation shows the effectiveness of our approach under different request distributions (e.g., Poisson, Pareto, and Constant inter-arrivals).