# EdgeShard: Efficient LLM Inference via Collaborative Edge Computing

arXiv: https://arxiv.org/abs/2405.14371 · HTML: https://mlsanity.com/paper/2405.14371/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 23 May 2024.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Large Language Models Empowered Autonomous Edge AI for Connected Intelligence](https://mlsanity.com/paper/2307.02779/) (2023)
- [Large Language Model Enhanced Multi-Agent Systems for 6G Communications](https://mlsanity.com/paper/2312.07850/) (2023)
- [Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge](https://mlsanity.com/paper/2312.05693/) (2023)
- [Squat: Quant Small Language Models on the Edge](https://mlsanity.com/paper/2402.10787/) (2024)
- [PrivateLoRA For Efficient Privacy Preserving LLM](https://mlsanity.com/paper/2311.14030/) (2023)
- [NetGPT: A Native-AI Network Architecture Beyond Provisioning Personalized Generative Services](https://mlsanity.com/paper/2307.06148/) (2023)
- [ENTS: An Edge-native Task Scheduling System for Collaborative Edge Computing](https://mlsanity.com/paper/2210.07842/) (2022)
- [EaaS: A Service-Oriented Edge Computing Framework Towards Distributed Intelligence](https://mlsanity.com/paper/2209.06613/) (2022)

## Built on by (most cited first, 7 of 7)

- [Towards Pervasive Distributed Agentic Generative AI -- A State of The Art](https://mlsanity.com/paper/2506.13324/) (2025)
- [EACO-RAG: Towards Distributed Tiered LLM Deployment using Edge-Assisted and Collaborative RAG with Adaptive Knowledge Update](https://mlsanity.com/paper/2410.20299/) (2024)
- [FollowUpBot: An LLM-Based Conversational Robot for Automatic Postoperative Follow-up](https://mlsanity.com/paper/2507.15502/) (2025)
- [SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference](https://mlsanity.com/paper/2604.21231/) (2026)
- [Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inference](https://mlsanity.com/paper/2608.28726/) (2026)
- [E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments](https://mlsanity.com/paper/2606.03770/) (2026)
- [Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment](https://mlsanity.com/paper/2608.15693/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown great potential in natural language processing and content generation. However, current LLMs heavily rely on cloud computing, leading to prolonged latency, high bandwidth cost, and privacy concerns. Edge computing is promising to address such concerns by deploying LLMs on edge devices, closer to data sources. Some works try to leverage model quantization to reduce the model size to fit the resource-constraint edge devices, but they lead to accuracy loss. Other works use cloud-edge collaboration, suffering from unstable network connections. In this work, we leverage collaborative edge computing to facilitate the collaboration among edge devices and cloud servers for jointly performing efficient LLM inference. We propose a general framework to partition the LLM model into shards and deploy on distributed devices. To achieve efficient LLM inference, we formulate an adaptive joint device selection and model partition problem and design an efficient dynamic programming algorithm to optimize the inference latency and throughput, respectively. Experiments of Llama2 serial models on a heterogeneous physical prototype demonstrate that EdgeShard achieves up to 50% latency reduction and 2x throughput improvement over baseline methods.