mlsanityData to 14 Oct 2026

EdgeServing: Deadline-Aware Multi-DNN Serving at the Edge

arXiv 2605.05527 ↗First posted 7 May 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

As edge computing expands, serving multiple deep neural network (DNN) models on a single shared GPU has become a common yet challenging scenario, where each scheduling decision affects the tail latency of all concurrent queues. Existing schedulers rely on local heuristics and fail to capture this global impact, while GPU spatial-sharing approaches sacrifice latency predictability. In this paper, we propose EdgeServing, a deadline-aware multi-DNN serving system for edge devices. EdgeServing adopts time-division GPU sharing with early-exit inference for high inference predictability, and introduces a stability score to quantify how each candidate scheduling decision impacts the future queue status. At runtime, it cohesively selects the model, exit point, and batch size to minimize predicted system-wide SLO impact. Experimental results on multiple hardware platforms show that EdgeServing consistently outperforms representative baselines in both SLO violation ratio and P95 latency, enabled by early-exit mechanism, which expands the scheduling action space under tight latency constraints.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 12 of its 22 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

12 of 22 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving8 Dec
  2. 2022You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model21 Nov

Other references · 10

  1. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  2. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  3. 2019Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computingcited by 228
  4. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  5. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  6. 2020SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloudcited by 112
  7. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  8. 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 70
  9. 2019A Survey on Edge Computing Systems and Toolscited by 17
  10. 2023Symphony: Optimized DNN Model Serving using Deferred Batch Schedulingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SparOA: Sparse and Operator-aware Hybrid Scheduling for Edge DNN Inference
  2. 2026Latency-Constrained DNN Architecture Learning for Edge Systems using Zerorized Batch Normalization
  3. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  4. 2026LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving
  5. 2025EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs
  6. 2026Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices
  7. 2026Edge-Inference Governors Need Memory-Clock State ↗
  8. 2026A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits