mlsanityData to 14 Oct 2026

Early-Exit meets Model-Distributed Inference at Edge Networks

arXiv 2408.05247 ↗First posted 8 Aug 2024cs.DC cs.AI cs.LGcited by 4

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Distributed inference techniques can be broadly classified into data-distributed and model-distributed schemes. In data-distributed inference (DDI), each worker carries the entire deep neural network (DNN) model but processes only a subset of the data. However, feeding the data to workers results in high communication costs, especially when the data is large. An emerging paradigm is model-distributed inference (MDI), where each worker carries only a subset of DNN layers. In MDI, a source device that has data processes a few layers of DNN and sends the output to a neighboring device, i.e., offloads the rest of the layers. This process ends when all layers are processed in a distributed manner. In this paper, we investigate the design and development of MDI with early-exit, which advocates that there is no need to process all the layers of a model for some data to reach the desired accuracy, i.e., we can exit the model without processing all the layers if target accuracy is reached. We design a framework MDI-Exit that adaptively determines early-exit and offloading policies as well as data admission at the source. Experimental results on a real-life testbed of NVIDIA Nano edge devices show that MDI-Exit processes more data when accuracy is fixed and results in higher accuracy for the fixed data rate.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 14 of its 24 references that are in the map. First posted on arXiv 8 Aug 2024.

Builds on

14 of 24 references.

References · 14

  1. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  2. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  3. 2021Dynamic Neural Networks: A Surveycited by 408
  4. 2018The Convergence of Sparsified Gradient Methodscited by 229
  5. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
  6. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 195
  7. 2017Adaptive Neural Networks for Efficient Inferencecited by 163
  8. 2015Compressing Neural Networks with the Hashing Trickcited by 150
  9. 2017Distributed Deep Neural Networks over the Cloud, the Edge and End Devicescited by 120
  10. 2018JointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Servicescited by 80
  11. 2015Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognitioncited by 67
  12. 2020Pruning Algorithms to Accelerate Convolutional Neural Networks for Edge Applications: A Surveycited by 26
2 more
  1. 2021Pipeline Parallelism for Inference on Heterogeneous Edge Computingcited by 13
  2. 2022Pruning Early Exit Networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2025Model-Distributed Inference for Large Language Models at the Edgecited by 8
  2. 2024Priority-Aware Model-Distributed Inference at Edge Networkscited by 2
  3. 2024Goal-oriented Communications based on Recursive Early Exit Neural Networkscited by 1
  4. 2026Position Paper: From Edge AI to Adaptive Edge AIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DistrEE: Distributed Early Exit of Deep Neural Network Inference on Edge Devices
  2. 2023DISCO: Distributed Inference with Sparse Communications
  3. 2024Distributed Inference on Mobile Edge and Cloud: An Early Exit based Clustering Approach
  4. 2021Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
  5. 2022Resource-Constrained Edge AI with Early Exit Prediction
  6. 2022Decentralized Low-Latency Collaborative Inference via Ensembles on the Edge
  7. 2023Adaptive Early Exiting for Collaborative Inference over Noisy Wireless Channels
  8. 2024Failure-Resilient Distributed Inference with Model Compression over Heterogeneous Edge Devices