# Edge-MultiAI: Multi-Tenancy of Latency-Sensitive Deep Learning Applications on Edge

arXiv: https://arxiv.org/abs/2211.07130 · HTML: https://mlsanity.com/paper/2211.07130/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 14 Nov 2022.

## Builds on (most cited first, 14 of 14)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing](https://mlsanity.com/paper/1905.10083/) (2019)
- [Convergence of Edge Computing and Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/1907.08349/) (2019)
- [Machine Learning at the Network Edge: A Survey](https://mlsanity.com/paper/1908.00080/) (2019)
- [Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence](https://mlsanity.com/paper/1909.00560/) (2019)
- [Adaptive Quantization for Deep Neural Network](https://mlsanity.com/paper/1712.01048/) (2017)
- [Towards Monocular Vision based Obstacle Avoidance through Deep Reinforcement Learning](https://mlsanity.com/paper/1706.09829/) (2017)
- [DeepIoT: Compressing Deep Neural Network Structures for Sensing Systems with a Compressor-Critic Framework](https://mlsanity.com/paper/1706.01215/) (2017)
- [Moving Deep Learning into Web Browser: How Far Can We Go?](https://mlsanity.com/paper/1901.09388/) (2019)
- [Analyzing the Performance of Smart Industry 4.0 Applications on Cloud Computing Systems](https://mlsanity.com/paper/2012.06054/) (2020)
- [SAED: Edge-Based Intelligence for Privacy-Preserving Enterprise Search on the Cloud](https://mlsanity.com/paper/2102.13367/) (2021)
- [Exploring the Impact of Virtualization on the Usability of the Deep Learning Applications](https://mlsanity.com/paper/2112.09780/) (2021)
- [Autonomous Task Dropping Mechanism to Achieve Robustness in Heterogeneous Computing Systems](https://mlsanity.com/paper/2005.11050/) (2020)
- [FELARE: Fair Scheduling of Machine Learning Tasks on Heterogeneous Edge Systems](https://mlsanity.com/paper/2206.00065/) (2022)

## Built on by (most cited first, 1 of 1)

- [Downsides of Smartness Across Edge-Cloud Continuum in Modern Industry](https://mlsanity.com/paper/2603.29289/) (2026)

## Abstract (arXiv)

Smart IoT-based systems often desire continuous execution of multiple latency-sensitive Deep Learning (DL) applications. The edge servers serve as the cornerstone of such IoT-based systems, however, their resource limitations hamper the continuous execution of multiple (multi-tenant) DL applications. The challenge is that, DL applications function based on bulky "neural network (NN) models" that cannot be simultaneously maintained in the limited memory space of the edge. Accordingly, the main contribution of this research is to overcome the memory contention challenge, thereby, meeting the latency constraints of the DL applications without compromising their inference accuracy. We propose an efficient NN model management framework, called Edge-MultiAI, that ushers the NN models of the DL applications into the edge memory such that the degree of multi-tenancy and the number of warm-starts are maximized. Edge-MultiAI leverages NN model compression techniques, such as model quantization, and dynamically loads NN models for DL applications to stimulate multi-tenancy on the edge server. We also devise a model management heuristic for Edge-MultiAI, called iWS-BFE, that functions based on the Bayesian theory to predict the inference requests for multi-tenant applications, and uses it to choose the appropriate NN models for loading, hence, increasing the number of warm-start inferences. We evaluate the efficacy and robustness of Edge-MultiAI under various configurations. The results reveal that Edge-MultiAI can stimulate the degree of multi-tenancy on the edge by at least 2X and increase the number of warm-starts by around 60% without any major loss on the inference accuracy of the applications.