mlsanityData to 7 Oct 2026

Design and Prototyping Distributed CNN Inference Acceleration in Edge Computing

arXiv 2211.13778 ↗First posted 24 Nov 2022cs.CV cs.DCcited by 0

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

For time-critical IoT applications using deep learning, inference acceleration through distributed computing is a promising approach to meet a stringent deadline. In this paper, we implement a working prototype of a new distributed inference acceleration method HALP using three raspberry Pi 4. HALP accelerates inference by designing a seamless collaboration among edge devices (EDs) in Edge Computing. We maximize the parallelization between communication and computation among the collaborative EDs by optimizing the task partitioning ratio based on the segment-based partitioning. Experimental results show that the distributed inference HALP achieves 1.7x inference acceleration for VGG-16. Then, we combine distributed inference with conventional neural network model compression by setting up different shrinking hyperparameters for MobileNet-V1. In this way, we can further accelerate inference but at the cost of inference accuracy loss. To strike a balance between latency and accuracy, we propose dynamic model selection to select a model which provides the highest accuracy within the latency constraint. It is shown that the model selection with distributed inference HALP can significantly improve service reliability compared to the conventional stand-alone computation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 5 of its 9 references that are in the map. First posted on arXiv 24 Nov 2022.

Builds on

5 of 9 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Distributed Deep Learning Inference Acceleration using Seamless Collaboration in Edge Computing22 Jul
  2. 2021Multi-Exit Vision Transformer for Dynamic Inference29 Jun

Other references · 3

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  3. 2022Dynamic Split Computing for Efficient Deep Edge Intelligencecited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Receptive Field-based Segmentation for Distributed CNN Inference Acceleration in Collaborative Edge Computing
  2. 2025Collaborative Inference Acceleration with Non-Penetrative Tensor Partitioning
  3. 2025DistrEE: Distributed Early Exit of Deep Neural Network Inference on Edge Devices
  4. 2024Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inference
  5. 2023DISCO: Distributed Inference with Sparse Communications
  6. 2024Collaborative Inference for Large Models with Task Offloading and Early Exiting
  7. 2026CANS: Accelerating Multiuser Collaborative Edge Inference via Cooperative Autodidactic NeuroSurgeon
  8. 2022Fault-Tolerant Collaborative Inference through the Edge-PRUNE Framework