mlsanityData to 14 Oct 2026

Collaborative Inference for Large Models with Task Offloading and Early Exiting

arXiv 2412.08284 ↗First posted 11 Dec 2024cs.DCcited by 0

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

In 5G smart cities, edge computing is employed to provide nearby computing services for end devices, and the large-scale models (e.g., GPT and LLaMA) can be deployed at the network edge to boost the service quality. However, due to the constraints of memory size and computing capacity, it is difficult to run these large-scale models on a single edge node. To meet the resource constraints, a large-scale model can be partitioned into multiple sub-models and deployed across multiple edge nodes. Then tasks are offloaded to the edge nodes for collaborative inference. Additionally, we incorporate the early exit mechanism to further accelerate inference. However, the heterogeneous system and dynamic environment will significantly affect the inference efficiency. To address these challenges, we theoretically analyze the coupled relationship between task offloading strategy and confidence thresholds, and develop a distributed algorithm, termed DTO-EE, based on the coupled relationship and convex optimization. DTO-EE enables each edge node to jointly optimize its offloading strategy and the confidence threshold, so as to achieve a promising trade-off between response delay and inference accuracy. The experimental results show that DTO-EE can reduce the average response delay by 21%-41% and improve the inference accuracy by 1%-4%, compared to the baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 5 of its 50 references that are in the map. First posted on arXiv 11 Dec 2024.

Builds on

5 of 50 references.

References · 5

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  4. 2024ParallelSFL: A Novel Split Federated Learning Framework Tackling Heterogeneity Issuescited by 10
  5. 2024Collaborative Satellite Computing through Adaptive DNN Task Splitting and Offloadingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Bayes-Split-Edge: Bayesian Optimization for Constrained Collaborative Inference in Wireless Edge Systems
  2. 2024Distributed Collaborative Inference System in Next-Generation Networks and Communication
  3. 2024EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
  4. 2022Design and Prototyping Distributed CNN Inference Acceleration in Edge Computing
  5. 2025Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
  6. 2025Cooperative Task Offloading through Asynchronous Deep Reinforcement Learning in Mobile Edge Computing for Future Networks
  7. 2024Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inference
  8. 2026Learning-Based Collaborative MEC for LLM Inference with Soft-Deadline Awareness via Transformer-Enhanced PPO