mlsanityData to 14 Oct 2026

E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments

arXiv 2606.03770 ↗First posted 2 Jun 2026cs.DC cs.AIcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large Language Models (LLMs) have become integral to modern applications, yet their deployment remains challenging. Beyond executing the models themselves, practical deployment must address cost efficiency, low latency, and optimal resource utilization. Conventional approaches typically assume that an entire model can be hosted on a single device, which does not hold in many real-world scenarios, particularly in Edge and Fog environments where device resources are constrained. In this paper, we introduce E2LLM, a framework designed to enable efficient LLM deployment in such resource limited settings. Rather than simply partitioning a single model across all available devices, E2LLM replicates the full model across multiple groups of devices (replicas) and applies model parallelism within each replica. Each replica is assigned a specialized role PREFILL or DECODER based on its efficiency in handling input and output tokens. This separation leverages the inherent differences between these two phases of LLM inference. To effectively organize devices, we utilize a Genetic Algorithm to form clusters that maximize system performance. Within each cluster, we apply Dynamic Programming to determine an optimal partitioning strategy that minimizes bottlenecks in model-parallel execution. Experimental results demonstrate that our approach adapts robustly to varying workloads, including scenarios with significant variation in input and output token lengths. Compared to the Splitwise baseline, E2LLM reduces average waiting time by over 50% under high-demand conditions

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 9 of its 26 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

9 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments13 Feb

Other references · 8

  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  2. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  3. 2020SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloudcited by 112
  4. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 50
  5. 2025HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environmentcited by 20
  6. 2021AppealNet: An Efficient and Highly-Accurate Edge/Cloud Collaborative Architecture for DNN Inferencecited by 17
  7. 2024EdgeShard: Efficient LLM Inference via Collaborative Edge Computingcited by 7
  8. 2025Synergistic Tensor and Pipeline Parallelismcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  2. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
  3. 2025EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices
  4. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  5. 2025MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices
  6. 2025Harli: SLO-Aware Co-location of LLM Inference and PEFT-based Finetuning on Model-as-a-Service Platforms
  7. 2025Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
  8. 2025Sometimes Painful but Certainly Promising: Feasibility and Trade-offs of Language Model Inference at the Edge