mlsanityData to 14 Oct 2026

LMEdge: QoS-Aware LLM Inference Orchestration on Edge Clusters

arXiv 2607.17175 ↗First posted 19 Jul 2026cs.DCcited by 1

Not on the map. Why?

Abstract (arXiv)

Large language model (LLM) services increasingly operate on edge infrastructure, enabling low-latency and privacy-preserving AI services. However, efficiently serving LLM requests across heterogeneous and resource-constrained edge devices require orchestration mechanisms that jointly determine model configuration (family, size, and quantization level) and execution placement while satisfying user- and system-level quality of service (QoS) requirements. This paper introduces LMEdge, a QoS-aware orchestration service that dynamically makes these decisions across heterogeneous edge devices. We formulate the problem as a binary integer linear programming (BILP) optimization that minimizes response time under accuracy, network, and resource constraints. To enable scalable online scheduling, we employ five lightweight machine learning (ML) models to predict query-specific latency, accuracy, resource usage, and response size for each model-size-quantization-device combination, and design a lightweight heuristic that approximates the BILP solution. We collect a comprehensive benchmarking dataset of over 59000 rows to train models and support reproducibility. Evaluation on a Kubernetes-based edge testbed with 57 instances and diverse query categories shows that LMEdge reduces latency, preserves accuracy, improves resource utilization, and increases serving ratio compared to two baselines.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 8 of its 14 references that are in the map. First posted on arXiv 19 Jul 2026.

Builds on

8 of 14 references.

References · 8

  1. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  2. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  3. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  4. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
  5. 2024Cost-Effective Online Multi-LLM Selection with Versatile Reward Modelscited by 39
  6. 2024OptLLM: Optimal Assignment of Queries to Large Language Modelscited by 16
  7. 2025Edge-First Language Model Inference: Models, Metrics, and Tradeoffscited by 13
  8. 2025Serving Long-Context LLMs at the Mobile Edge: Test-Time Reinforcement Learning-based Model Caching and Inference Offloadingcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026DRLM: Deep Reinforcement Learning-Based LLM Query Orchestration in Edge Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DILEMMA: Joint LLM Quantization and Distributed LLM Inference Over Edge Computing Systems
  2. 2024EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
  3. 2025Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
  4. 2025Niyama : Breaking the Silos of LLM Inference Serving
  5. 2026E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments
  6. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
  7. 2025Recursive Offloading for LLM Serving in Multi-tier Networks
  8. 2026A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware