# Online Learning for Orchestration of Inference in Multi-User End-Edge-Cloud Networks

arXiv: https://arxiv.org/abs/2202.10541 · HTML: https://mlsanity.com/paper/2202.10541/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 55 references that are in the map.
- First posted on arXiv 21 Feb 2022.

## Builds on (most cited first, 18 of 18)

- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Distributed Deep Neural Networks over the Cloud, the Edge and End Devices](https://mlsanity.com/paper/1709.01921/) (2017)
- [Wireless Network Intelligence at the Edge](https://mlsanity.com/paper/1812.02858/) (2018)
- [JointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Services](https://mlsanity.com/paper/1801.08618/) (2018)
- [Mobile Edge Computing: A Survey on Architecture and Computation Offloading](https://mlsanity.com/paper/1702.05309/) (2017)
- [Optimized Computation Offloading Performance in Virtual Edge Computing Systems via Deep Reinforcement Learning](https://mlsanity.com/paper/1805.06146/) (2018)
- [Learning-Based Computation Offloading for IoT Devices with Energy Harvesting](https://mlsanity.com/paper/1712.08768/) (2017)
- [All One Needs to Know about Fog Computing and Related Edge Computing Paradigms: A Complete Survey](https://mlsanity.com/paper/1808.05283/) (2018)
- [AdaDeep: A Usage-Driven, Automated Deep Model Compression Framework for Enabling Ubiquitous Intelligent Mobiles](https://mlsanity.com/paper/2006.04432/) (2020)
- [Embedded Binarized Neural Networks](https://mlsanity.com/paper/1709.02260/) (2017)
- [Decentralized Computation Offloading for Multi-User Mobile Edge Computing: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1812.07394/) (2018)
- [Online Learning for Offloading and Autoscaling in Renewable-Powered Mobile Edge Computing](https://mlsanity.com/paper/1609.05087/) (2016)

## Built on by (most cited first, 5 of 5)

- [Edge-Cloud Collaborative Computing on Distributed Intelligence and Model Optimization: A Survey](https://mlsanity.com/paper/2505.01821/) (2025)
- [Edge-centric Optimization of Multi-modal ML-driven eHealth Applications](https://mlsanity.com/paper/2208.02597/) (2022)
- [Adaptive Workload Distribution for Accuracy-aware DNN Inference on Collaborative Edge Platforms](https://mlsanity.com/paper/2310.10157/) (2023)
- [Positioning Fog Computing for Smart Manufacturing](https://mlsanity.com/paper/2205.10860/) (2022)
- [Characterizing Accuracy Trade-offs of EEG Applications on Embedded HMPs](https://mlsanity.com/paper/2402.09867/) (2024)

## Abstract (arXiv)

Deep-learning-based intelligent services have become prevalent in cyber-physical applications including smart cities and health-care. Deploying deep-learning-based intelligence near the end-user enhances privacy protection, responsiveness, and reliability. Resource-constrained end-devices must be carefully managed in order to meet the latency and energy requirements of computationally-intensive deep learning services. Collaborative end-edge-cloud computing for deep learning provides a range of performance and efficiency that can address application requirements through computation offloading. The decision to offload computation is a communication-computation co-optimization problem that varies with both system parameters (e.g., network condition) and workload characteristics (e.g., inputs). On the other hand, deep learning model optimization provides another source of tradeoff between latency and model accuracy. An end-to-end decision-making solution that considers such computation-communication problem is required to synergistically find the optimal offloading policy and model for deep learning services. To this end, we propose a reinforcement-learning-based computation offloading solution that learns optimal offloading policy considering deep learning model selection techniques to minimize response time while providing sufficient accuracy. We demonstrate the effectiveness of our solution for edge devices in an end-edge-cloud system and evaluate with a real-setup implementation using multiple AWS and ARM core configurations. Our solution provides 35% speedup in the average response time compared to the state-of-the-art with less than 0.9% accuracy reduction, demonstrating the promise of our online learning framework for orchestrating DL inference in end-edge-cloud systems.