# Priority-Aware Model-Distributed Inference at Edge Networks

arXiv: https://arxiv.org/abs/2412.12371 · HTML: https://mlsanity.com/paper/2412.12371/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 16 Dec 2024.

## Builds on (most cited first, 19 of 19)

- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [The Convergence of Sparsified Gradient Methods](https://mlsanity.com/paper/1809.10505/) (2018)
- [Gradient Sparsification for Communication-Efficient Distributed Optimization](https://mlsanity.com/paper/1710.09854/) (2017)
- [Compressing Deep Convolutional Networks using Vector Quantization](https://mlsanity.com/paper/1412.6115/) (2014)
- [Adaptive Neural Networks for Efficient Inference](https://mlsanity.com/paper/1702.07811/) (2017)
- [Compressing Neural Networks with the Hashing Trick](https://mlsanity.com/paper/1504.04788/) (2015)
- [Distributed Deep Neural Networks over the Cloud, the Edge and End Devices](https://mlsanity.com/paper/1709.01921/) (2017)
- [PipeDream: Fast and Efficient Pipeline Parallel DNN Training](https://mlsanity.com/paper/1806.03377/) (2018)
- [JointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Services](https://mlsanity.com/paper/1801.08618/) (2018)
- [Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform](https://mlsanity.com/paper/1809.02839/) (2018)
- [Pruning Algorithms to Accelerate Convolutional Neural Networks for Edge Applications: A Survey](https://mlsanity.com/paper/2005.04275/) (2020)
- [Improving Device-Edge Cooperative Inference of Deep Learning via 2-Step Pruning](https://mlsanity.com/paper/1903.03472/) (2019)
- [Pipeline Parallelism for Inference on Heterogeneous Edge Computing](https://mlsanity.com/paper/2110.14895/) (2021)
- [Colosseum: Large-Scale Wireless Experimentation Through Hardware-in-the-Loop Network Emulation](https://mlsanity.com/paper/2110.10617/) (2021)
- [Pruning Early Exit Networks](https://mlsanity.com/paper/2207.03644/) (2022)
- [Early-Exit meets Model-Distributed Inference at Edge Networks](https://mlsanity.com/paper/2408.05247/) (2024)

## Built on by (most cited first, 2 of 2)

- [Model-Distributed Inference for Large Language Models at the Edge](https://mlsanity.com/paper/2505.18164/) (2025)
- [Time, Causality, and Observability Failures in Distributed AI Inference Systems](https://mlsanity.com/paper/2604.21361/) (2026)

## Abstract (arXiv)

Distributed inference techniques can be broadly classified into data-distributed and model-distributed schemes. In data-distributed inference (DDI), each worker carries the entire Machine Learning (ML) model but processes only a subset of the data. However, feeding the data to workers results in high communication costs, especially when the data is large. An emerging paradigm is model-distributed inference (MDI), where each worker carries only a subset of ML layers. In MDI, a source device that has data processes a few layers of ML model and sends the output to a neighboring device, i.e., offloads the rest of the layers. This process ends when all layers are processed in a distributed manner. In this paper, we investigate the design and development of MDI when multiple data sources co-exist. We consider that each data source has a different importance and, hence, a priority. We formulate and solve a priority-aware model allocation optimization problem. Based on the structure of the optimal solution, we design a practical Priority-Aware Model- Distributed Inference (PA-MDI) algorithm that determines model allocation and distribution over devices by taking into account the priorities of different sources. Experiments were conducted on a real-life testbed of NVIDIA Jetson Xavier and Nano edge devices as well as in the Colosseum testbed with ResNet-50, ResNet- 56, and GPT-2 models. The experimental results show that PA-MDI performs priority-aware model allocation successfully while reducing the inference time as compared to baselines.