mlsanityData to 7 Oct 2026

Collaborative Inference Acceleration with Non-Penetrative Tensor Partitioning

arXiv 2501.04489 ↗First posted 8 Jan 2025cs.DCcited by 0

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

The inference of large-sized images on Internet of Things (IoT) devices is commonly hindered by limited resources, while there are often stringent latency requirements for Deep Neural Network (DNN) inference. Currently, this problem is generally addressed by collaborative inference, where the large-sized image is partitioned into multiple tiles, and each tile is assigned to an IoT device for processing. However, since significant latency will be incurred due to the communication overhead caused by tile sharing, the existing collaborative inference strategy is inefficient for convolutional computation, which is indispensable for any DNN. To reduce it, we propose Non-Penetrative Tensor Partitioning (NPTP), a fine-grained tensor partitioning method that reduces the communication latency by minimizing the communication load of tiles shared, thereby reducing inference latency. We evaluate NPTP with four widely-adopted DNN models. Experimental results demonstrate that NPTP achieves a 1.44-1.68x inference speedup relative to CoEdge, a state-of-the-art (SOTA) collaborative inference algorithm.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 9 of its 26 references that are in the map. First posted on arXiv 8 Jan 2025.

Builds on

9 of 26 references.

References · 9

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2020MCUNet: Tiny Deep Learning on IoT Devicescited by 269
  3. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  4. 2021MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learningcited by 87
  5. 2020CoEdge: Cooperative DNN Inference with Adaptive Workload Partitioning over Heterogeneous Edge Devicescited by 65
  6. 2022A Survey on Collaborative DNN Inference for Edge Intelligencecited by 23
  7. 2022On Optimizing the Communication of Model Parallelismcited by 19
  8. 2022Collaborative Inference for AI-Empowered IoT Devicescited by 14
  9. 2022Accuracy-Guaranteed Collaborative DNN Inference in Industrial IoT via Deep Reinforcement Learningcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Design and Prototyping Distributed CNN Inference Acceleration in Edge Computing
  2. 2022Distributed Deep Learning Inference Acceleration using Seamless Collaboration in Edge Computing
  3. 2022Receptive Field-based Segmentation for Distributed CNN Inference Acceleration in Collaborative Edge Computing
  4. 2025Hetero-SplitEE: Split Learning of Neural Networks with Early Exits for Heterogeneous IoT Devices
  5. 2026CANS: Accelerating Multiuser Collaborative Edge Inference via Cooperative Autodidactic NeuroSurgeon
  6. 2024Federated Learning for Collaborative Inference Systems: The Case of Early Exit Networks
  7. 2025DistrEE: Distributed Early Exit of Deep Neural Network Inference on Edge Devices
  8. 2023Adaptive Early Exiting for Collaborative Inference over Noisy Wireless Channels