mlsanityData to 7 Oct 2026

Distributed Inference on Mobile Edge and Cloud: An Early Exit based Clustering Approach

arXiv 2410.05338 ↗First posted 6 Oct 2024cs.LG cs.AI cs.DCcited by 3

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Recent advances in Deep Neural Networks (DNNs) have demonstrated outstanding performance across various domains. However, their large size is a challenge for deployment on resource-constrained devices such as mobile, edge, and IoT platforms. To overcome this, a distributed inference setup can be used where a small-sized DNN (initial few layers) can be deployed on mobile, a bigger version on the edge, and the full-fledged, on the cloud. A sample that has low complexity (easy) could be then inferred on mobile, that has moderate complexity (medium) on edge, and higher complexity (hard) on the cloud. As the complexity of each sample is not known beforehand, the following question arises in distributed inference: how to decide complexity so that it is processed by enough layers of DNNs. We develop a novel approach named DIMEE that utilizes Early Exit (EE) strategies developed to minimize inference latency in DNNs. DIMEE aims to improve the accuracy, taking into account the offloading cost from mobile to edge/cloud. Experimental validation on GLUE datasets, encompassing various NLP tasks, shows that our method significantly reduces the inference cost (> 43%) while maintaining a minimal drop in accuracy (< 0.3%) compared to the case where all the inference is made in cloud.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 40 references that are in the map. First posted on arXiv 6 Oct 2024.

Builds on

26 of 40 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024CEEBERT: Cross-Domain Inference in Early Exit BERT23 May
  2. 2023SplitEE: Early Exit in Deep Neural Networks with Split Computing17 Sep
  3. 2021Early-exit deep neural networks for distorted images: providing an efficient edge offloading20 Aug
  4. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges8 Mar

Other references · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  6. 2019Are Sixteen Heads Really Better than One?cited by 803
  7. 2020A Survey on Visual Transformercited by 653
  8. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  9. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  10. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  11. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  12. 2021I-BERT: Integer-only BERT Quantizationcited by 250
10 more
  1. 2019Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computingcited by 150
  2. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146
  3. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  4. 2020SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloudcited by 109
  5. 2019BottleNet++: An End-to-End Approach for Feature Compression in Device-Edge Co-Inference Systemscited by 94
  6. 2019BottleNet: A Deep Learning Architecture for Intelligent Mobile Cloud Computing Servicescited by 67
  7. 2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
  8. 2020Neural Compression and Filtering for Edge-assisted Real-time Object Detection in Challenged Networkscited by 28
  9. 2022Unsupervised Early Exit in DNNs with Multiple Exitscited by 10
  10. 2024I-SplitEE: Image classification in Split Computing DNNs with Early Exitscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and Reliabilitycited by 3
  2. 2025Know What You Don't Know: Selective Prediction for Early Exit DNNscited by 0
  3. 2026SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DistrEE: Distributed Early Exit of Deep Neural Network Inference on Edge Devices
  2. 2024Early-Exit meets Model-Distributed Inference at Edge Networks
  3. 2023DISCO: Distributed Inference with Sparse Communications
  4. 2024Collaborative Inference for Large Models with Task Offloading and Early Exiting
  5. 2023Adaptive DNN Surgery for Selfish Inference Acceleration with On-demand Edge Resource
  6. 2024A Converting Autoencoder Toward Low-latency and Energy-efficient DNN Inference at the Edge
  7. 2023Adaptive Early Exiting for Collaborative Inference over Noisy Wireless Channels
  8. 2022Design and Prototyping Distributed CNN Inference Acceleration in Edge Computing