# Multi-user Co-inference with Batch Processing Capable Edge Server

arXiv: https://arxiv.org/abs/2206.06304 · HTML: https://mlsanity.com/paper/2206.06304/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 3 Jun 2022.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Energy Efficient Federated Learning Over Wireless Communication Networks](https://mlsanity.com/paper/1911.02417/) (2019)
- [Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing](https://mlsanity.com/paper/1910.05316/) (2019)
- [Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach](https://mlsanity.com/paper/2102.04170/) (2021)
- [Communication-Computation Trade-Off in Resource-Constrained Edge Inference](https://mlsanity.com/paper/2006.02166/) (2020)
- [Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space](https://mlsanity.com/paper/1903.01344/) (2019)
- [Continuous-Discrete Reinforcement Learning for Hybrid Control in Robotics](https://mlsanity.com/paper/2001.00449/) (2020)
- [Edge-Host Partitioning of Deep Neural Networks with Feature Space Encoding for Resource-Constrained Internet-of-Things Platforms](https://mlsanity.com/paper/1802.03835/) (2018)
- [Accelerating DNN Training in Wireless Federated Edge Learning Systems](https://mlsanity.com/paper/1905.09712/) (2019)
- [Communicate to Learn at the Edge](https://mlsanity.com/paper/2009.13269/) (2020)
- [Improving Device-Edge Cooperative Inference of Deep Learning via 2-Step Pruning](https://mlsanity.com/paper/1903.03472/) (2019)
- [LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inference](https://mlsanity.com/paper/2010.13103/) (2020)
- [Branchy-GNN: a Device-Edge Co-Inference Framework for Efficient Point Cloud Processing](https://mlsanity.com/paper/2011.02422/) (2020)
- [Latency and Throughput Characterization of Convolutional Neural Networks for Mobile Computer Vision](https://mlsanity.com/paper/1803.09492/) (2018)
- [Joint Multi-User DNN Partitioning and Computational Resource Allocation for Collaborative Edge Intelligence](https://mlsanity.com/paper/2007.09072/) (2020)
- [Edge Learning with Timeliness Constraints: Challenges and Solutions](https://mlsanity.com/paper/2009.11065/) (2020)

## Built on by (most cited first, 11 of 11)

- [DVFS-Aware DNN Inference on GPUs: Latency Modeling and Performance Analysis](https://mlsanity.com/paper/2502.06295/) (2025)
- [SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms](https://mlsanity.com/paper/2301.12865/) (2023)
- [Robust DNN Partitioning and Resource Allocation Under Uncertain Inference Time](https://mlsanity.com/paper/2503.21476/) (2025)
- [PartialLoading: User Scheduling and Bandwidth Allocation for Parameter-sharing Edge Inference](https://mlsanity.com/paper/2503.22982/) (2025)
- [Joint Memory Frequency and Computing Frequency Scaling for Energy-efficient DNN Inference](https://mlsanity.com/paper/2509.17970/) (2025)
- [SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Services](https://mlsanity.com/paper/2501.02181/) (2025)
- [Towards A Flexible Accuracy-Oriented Deep Learning Module Inference Latency Prediction Framework for Adaptive Optimization Algorithms](https://mlsanity.com/paper/2312.06440/) (2023)
- [Hierarchical Online-Scheduling for Energy-Efficient Split Inference with Progressive Transmission](https://mlsanity.com/paper/2601.08135/) (2026)
- [BALANCE: Hybrid Autoregressive-Speculative LLM Inference at the Network Edge](https://mlsanity.com/paper/2608.05926/) (2026)
- [Joint Optimization of Memory and Computing Frequency for Energy-Efficient DNN Inference](https://mlsanity.com/paper/2608.13863/) (2026)
- [MANE: A Multi-Path Adaptive Network for Edge Onloading of Deep Neural Networks](https://mlsanity.com/paper/2609.14660/) (2026)

## Abstract (arXiv)

Graphics processing units (GPUs) can improve deep neural network inference throughput via batch processing, where multiple tasks are concurrently processed. We focus on novel scenarios that the energy-constrained mobile devices offload inference tasks to an edge server with GPU. The inference task is partitioned into sub-tasks for a finer granularity of offloading and scheduling, and the user energy consumption minimization problem under inference latency constraints is investigated. To deal with the coupled offloading and scheduling introduced by concurrent batch processing, we first consider an offline problem with a constant edge inference latency and the same latency constraint. It is proven that optimizing the offloading policy of each user independently and aggregating all the same sub-tasks in one batch is optimal, and thus the independent partitioning and same sub-task aggregating (IP-SSA) algorithm is inspired. Further, the optimal grouping (OG) algorithm is proposed to optimally group tasks when the latency constraints are different. Finally, when future task arrivals cannot be precisely predicted, a deep deterministic policy gradient (DDPG) agent is trained to call OG. Experiments show that IP-SSA reduces up to 94.9% user energy consumption in the offline setting, while DDPG-OG outperforms DDPG-IP-SSA by up to 8.92% in the online setting.