# Pipeline Parallelism for Inference on Heterogeneous Edge Computing

arXiv: https://arxiv.org/abs/2110.14895 · HTML: https://mlsanity.com/paper/2110.14895/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 55 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases](https://mlsanity.com/paper/2103.10697/) (2021)
- [HAT: Hardware-Aware Transformers for Efficient Natural Language Processing](https://mlsanity.com/paper/2005.14187/) (2020)
- [Memory-Efficient Pipeline-Parallel DNN Training](https://mlsanity.com/paper/2006.09503/) (2020)
- [Lite Transformer with Long-Short Range Attention](https://mlsanity.com/paper/2004.11886/) (2020)
- [Pixel-Adaptive Convolutional Neural Networks](https://mlsanity.com/paper/1904.05373/) (2019)
- [A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers](https://mlsanity.com/paper/1804.03294/) (2018)
- [JointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Services](https://mlsanity.com/paper/1801.08618/) (2018)
- [Towards Efficient Model Compression via Learned Global Ranking](https://mlsanity.com/paper/1904.12368/) (2019)
- [TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models](https://mlsanity.com/paper/2102.07988/) (2021)
- [EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference](https://mlsanity.com/paper/2011.14203/) (2020)
- [PipeMare: Asynchronous Pipeline Parallel DNN Training](https://mlsanity.com/paper/1910.05124/) (2019)
- [HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism](https://mlsanity.com/paper/2005.14038/) (2020)
- [A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNs](https://mlsanity.com/paper/2011.03083/) (2020)
- [torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models](https://mlsanity.com/paper/2004.09910/) (2020)
- [PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers](https://mlsanity.com/paper/2102.03161/) (2021)
- [Vehicular Edge Computing and Networking: A Survey](https://mlsanity.com/paper/1908.06849/) (2019)
- [AttentionLite: Towards Efficient Self-Attention Models for Vision](https://mlsanity.com/paper/2101.05216/) (2020)

## Built on by (most cited first, 13 of 13)

- [Vision-Language Models for Edge Networks: A Comprehensive Survey](https://mlsanity.com/paper/2502.07855/) (2025)
- [HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environments](https://mlsanity.com/paper/2111.10635/) (2021)
- [LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization](https://mlsanity.com/paper/2403.01136/) (2024)
- [STI: Turbocharge NLP Inference at the Edge via Elastic Pipelining](https://mlsanity.com/paper/2207.05022/) (2022)
- [KV Cache Transform Coding for Compact Storage in LLM Inference](https://mlsanity.com/paper/2511.01815/) (2025)
- [Model-Distributed Inference for Large Language Models at the Edge](https://mlsanity.com/paper/2505.18164/) (2025)
- [Early-Exit meets Model-Distributed Inference at Edge Networks](https://mlsanity.com/paper/2408.05247/) (2024)
- [SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decoding](https://mlsanity.com/paper/2504.04104/) (2025)
- [Priority-Aware Model-Distributed Inference at Edge Networks](https://mlsanity.com/paper/2412.12371/) (2024)
- [Privacy-Preserving Hierarchical Model-Distributed Inference](https://mlsanity.com/paper/2407.18353/) (2024)
- [Go With The Flow: Churn-Tolerant Decentralized Training of Large Language Models](https://mlsanity.com/paper/2509.21221/) (2025)
- [UNIFERENCE: A Discrete Event Simulation Framework for Developing Distributed AI Models](https://mlsanity.com/paper/2603.26469/) (2026)
- [Integrity of peer-to-peer distributed LLM inference under malicious nodes](https://mlsanity.com/paper/2607.19490/) (2026)

## Abstract (arXiv)

Deep neural networks with large model sizes achieve state-of-the-art results for tasks in computer vision (CV) and natural language processing (NLP). However, these large-scale models are too compute- or memory-intensive for resource-constrained edge devices. Prior works on parallel and distributed execution primarily focus on training -- rather than inference -- using homogeneous accelerators in data centers. We propose EdgePipe, a distributed framework for edge systems that uses pipeline parallelism to both speed up inference and enable running larger (and more accurate) models that otherwise cannot fit on single edge devices. EdgePipe achieves these results by using an optimal partition strategy that considers heterogeneity in compute, memory, and network bandwidth. Our empirical evaluation demonstrates that EdgePipe achieves 10.59× and 11.88× speedup using 16 edge devices for the ViT-Large and ViT-Huge models, respectively, with no accuracy loss. Similarly, EdgePipe improves ViT-Huge throughput by 3.93× over a 4-node baseline using 16 edge devices, which independently cannot fit the model in memory. Finally, we show up to 4.16× throughput improvement over the state-of-the-art PipeDream when using a heterogeneous set of devices.