# Distributed Inference on Mobile Edge and Cloud: An Early Exit based Clustering Approach

arXiv: https://arxiv.org/abs/2410.05338 · HTML: https://mlsanity.com/paper/2410.05338/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 6 Oct 2024.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing](https://mlsanity.com/paper/1910.05316/) (2019)
- [TernaryBERT: Distillation-aware Ultra-low Bit BERT](https://mlsanity.com/paper/2009.12812/) (2020)
- [Resolution Adaptive Networks for Efficient Inference](https://mlsanity.com/paper/2003.07326/) (2020)
- [Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges](https://mlsanity.com/paper/2103.04505/) (2021)
- [SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud](https://mlsanity.com/paper/2008.06402/) (2020)
- [BottleNet++: An End-to-End Approach for Feature Compression in Device-Edge Co-Inference Systems](https://mlsanity.com/paper/1910.14315/) (2019)
- [BottleNet: A Deep Learning Architecture for Intelligent Mobile Cloud Computing Services](https://mlsanity.com/paper/1902.01000/) (2019)
- [Towards Efficient NLP: A Standard Evaluation and A Strong Baseline](https://mlsanity.com/paper/2110.07038/) (2021)
- [Neural Compression and Filtering for Edge-assisted Real-time Object Detection in Challenged Networks](https://mlsanity.com/paper/2007.15818/) (2020)
- [SplitEE: Early Exit in Deep Neural Networks with Split Computing](https://mlsanity.com/paper/2309.09195/) (2023)
- [CEEBERT: Cross-Domain Inference in Early Exit BERT](https://mlsanity.com/paper/2405.15039/) (2024)
- [Unsupervised Early Exit in DNNs with Multiple Exits](https://mlsanity.com/paper/2209.09480/) (2022)
- [I-SplitEE: Image classification in Split Computing DNNs with Early Exits](https://mlsanity.com/paper/2401.10541/) (2024)
- [Early-exit deep neural networks for distorted images: providing an efficient edge offloading](https://mlsanity.com/paper/2108.09343/) (2021)

## Built on by (most cited first, 3 of 3)

- [Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and Reliability](https://mlsanity.com/paper/2509.23666/) (2025)
- [Know What You Don't Know: Selective Prediction for Early Exit DNNs](https://mlsanity.com/paper/2509.11520/) (2025)
- [SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference](https://mlsanity.com/paper/2608.13076/) (2026)

## Abstract (arXiv)

Recent advances in Deep Neural Networks (DNNs) have demonstrated outstanding performance across various domains. However, their large size is a challenge for deployment on resource-constrained devices such as mobile, edge, and IoT platforms. To overcome this, a distributed inference setup can be used where a small-sized DNN (initial few layers) can be deployed on mobile, a bigger version on the edge, and the full-fledged, on the cloud. A sample that has low complexity (easy) could be then inferred on mobile, that has moderate complexity (medium) on edge, and higher complexity (hard) on the cloud. As the complexity of each sample is not known beforehand, the following question arises in distributed inference: how to decide complexity so that it is processed by enough layers of DNNs. We develop a novel approach named DIMEE that utilizes Early Exit (EE) strategies developed to minimize inference latency in DNNs. DIMEE aims to improve the accuracy, taking into account the offloading cost from mobile to edge/cloud. Experimental validation on GLUE datasets, encompassing various NLP tasks, shows that our method significantly reduces the inference cost (> 43%) while maintaining a minimal drop in accuracy (< 0.3%) compared to the case where all the inference is made in cloud.