# EdgeServing: Deadline-Aware Multi-DNN Serving at the Edge

arXiv: https://arxiv.org/abs/2605.05527 · HTML: https://mlsanity.com/paper/2605.05527/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 7 May 2026.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 12 of 12)

- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing](https://mlsanity.com/paper/1905.10083/) (2019)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Serving DNNs like Clockwork: Performance Predictability from the Bottom Up](https://mlsanity.com/paper/2006.02464/) (2020)
- [SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud](https://mlsanity.com/paper/2008.06402/) (2020)
- [TensorFlow-Serving: Flexible, High-Performance ML Serving](https://mlsanity.com/paper/1712.06139/) (2017)
- [EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference](https://mlsanity.com/paper/2011.14203/) (2020)
- [You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model](https://mlsanity.com/paper/2211.11152/) (2022)
- [A Survey on Edge Computing Systems and Tools](https://mlsanity.com/paper/1911.02794/) (2019)
- [Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving](https://mlsanity.com/paper/2312.05385/) (2023)
- [Symphony: Optimized DNN Model Serving using Deferred Batch Scheduling](https://mlsanity.com/paper/2308.07470/) (2023)

## Abstract (arXiv)

As edge computing expands, serving multiple deep neural network (DNN) models on a single shared GPU has become a common yet challenging scenario, where each scheduling decision affects the tail latency of all concurrent queues. Existing schedulers rely on local heuristics and fail to capture this global impact, while GPU spatial-sharing approaches sacrifice latency predictability. In this paper, we propose EdgeServing, a deadline-aware multi-DNN serving system for edge devices. EdgeServing adopts time-division GPU sharing with early-exit inference for high inference predictability, and introduces a stability score to quantify how each candidate scheduling decision impacts the future queue status. At runtime, it cohesively selects the model, exit point, and batch size to minimize predicted system-wide SLO impact. Experimental results on multiple hardware platforms show that EdgeServing consistently outperforms representative baselines in both SLO violation ratio and P95 latency, enabled by early-exit mechanism, which expands the scheduling action space under tight latency constraints.