# Model-Distributed Inference for Large Language Models at the Edge

arXiv: https://arxiv.org/abs/2505.18164 · HTML: https://mlsanity.com/paper/2505.18164/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 13 May 2025.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [Atom: Low-bit Quantization for Efficient and Accurate LLM Serving](https://mlsanity.com/paper/2310.19102/) (2023)
- [Compressing LLMs: The Truth is Rarely Pure and Never Simple](https://mlsanity.com/paper/2310.01382/) (2023)
- [Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform](https://mlsanity.com/paper/1809.02839/) (2018)
- [Tesseract: Parallelize the Tensor Parallelism Efficiently](https://mlsanity.com/paper/2105.14500/) (2021)
- [Pipeline Parallelism for Inference on Heterogeneous Edge Computing](https://mlsanity.com/paper/2110.14895/) (2021)
- [Early-Exit meets Model-Distributed Inference at Edge Networks](https://mlsanity.com/paper/2408.05247/) (2024)
- [Priority-Aware Model-Distributed Inference at Edge Networks](https://mlsanity.com/paper/2412.12371/) (2024)

## Built on by (most cited first, 8 of 8)

- [Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets](https://mlsanity.com/paper/2608.19147/) (2026)
- [Military AI Cyber Agents (MAICAs) Constitute a Global Threat to Critical Infrastructure](https://mlsanity.com/paper/2506.12094/) (2025)
- [Cost and Accuracy of Long-Term Memory in Distributed Multi-Agent Systems Based on Large Language Models](https://mlsanity.com/paper/2601.07978/) (2026)
- [Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networks](https://mlsanity.com/paper/2511.02647/) (2025)
- [Privacy-Aware Split Inference with Speculative Decoding for Large Language Models over Wide-Area Networks](https://mlsanity.com/paper/2602.16760/) (2026)
- [CIDERS: Cloud-Edge LLM Collaborative Learning via Accelerating Personalized Bilevel Optimization](https://mlsanity.com/paper/2609.15664/) (2026)
- [TopoCompress: Topology Aware Token Compression Algorithm for Distributed Edge MoE Inference](https://mlsanity.com/paper/2609.26061/) (2026)
- [Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructure](https://mlsanity.com/paper/2609.38697/) (2026)

## Abstract (arXiv)

We introduce Model-Distributed Inference for Large-Language Models (MDI-LLM), a novel framework designed to facilitate the deployment of state-of-the-art large-language models (LLMs) across low-power devices at the edge. This is accomplished by dividing the model into multiple partitions, which are then assigned to different devices/nodes within the network. These nodes exchange intermediate activation vectors via device-to-device links, enabling collaborative computation. To enhance the efficiency of this process, we propose the "recurrent pipeline parallelism" technique, which reduces idle time on each device and facilitates parallel inference during the generation of multiple text sequences. By leveraging the combined computational resources of multiple edge devices, MDI-LLM enables the deployment of LLMs that exceed the memory capacity of individual devices, making it possible to perform inference on low-cost hardware. Furthermore, as the number of participating devices increases, MDI-LLM boosts token generation throughput and reduces memory consumption per device.