# Memory- and Communication-Aware Model Compression for Distributed Deep Learning Inference on IoT

arXiv: https://arxiv.org/abs/1907.11804 · HTML: https://mlsanity.com/paper/1907.11804/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2019.

## Built on by (most cited first, 13 of 13)

- [Packet-Loss-Tolerant Split Inference for Delay-Sensitive Deep Learning in Lossy Wireless Networks](https://mlsanity.com/paper/2104.13629/) (2021)
- [Compacting Deep Neural Networks for Internet of Things: Methods and Applications](https://mlsanity.com/paper/2103.11083/) (2021)
- [An improved LogNNet classifier for IoT application](https://mlsanity.com/paper/2105.14412/) (2021)
- [Efficient Computer Vision on Edge Devices with Pipeline-Parallel Hierarchical Neural Networks](https://mlsanity.com/paper/2109.13356/) (2021)
- [Knowledge Distillation: Enhancing Neural Network Compression with Integrated Gradients](https://mlsanity.com/paper/2503.13008/) (2025)
- [Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learning](https://mlsanity.com/paper/2011.14266/) (2020)
- [Failure-Resilient Distributed Inference with Model Compression over Heterogeneous Edge Devices](https://mlsanity.com/paper/2406.14185/) (2024)
- [DistrEE: Distributed Early Exit of Deep Neural Network Inference on Edge Devices](https://mlsanity.com/paper/2502.15735/) (2025)
- [ParaDiS: Parallelly Distributable Slimmable Neural Networks](https://mlsanity.com/paper/2110.02724/) (2021)
- [Bandwidth-efficient distributed neural network architectures with application to body sensor networks](https://mlsanity.com/paper/2210.07750/) (2022)
- [A Theory of I/O-Efficient Sparse Neural Network Inference](https://mlsanity.com/paper/2301.01048/) (2023)
- [Model compression using knowledge distillation with integrated gradients](https://mlsanity.com/paper/2506.14440/) (2025)
- [Uncertainty Makes It Stable: Curiosity-Driven Quantized Mixture-of-Experts](https://mlsanity.com/paper/2511.11743/) (2025)

## Abstract (arXiv)

Model compression has emerged as an important area of research for deploying deep learning models on Internet-of-Things (IoT). However, for extremely memory-constrained scenarios, even the compressed models cannot fit within the memory of a single device and, as a result, must be distributed across multiple devices. This leads to a distributed inference paradigm in which memory and communication costs represent a major bottleneck. Yet, existing model compression techniques are not communication-aware. Therefore, we propose Network of Neural Networks (NoNN), a new distributed IoT learning paradigm that compresses a large pretrained 'teacher' deep network into several disjoint and highly-compressed 'student' modules, without loss of accuracy. Moreover, we propose a network science-based knowledge partitioning algorithm for the teacher model, and then train individual students on the resulting disjoint partitions. Extensive experimentation on five image classification datasets, for user-defined memory/performance budgets, show that NoNN achieves higher accuracy than several baselines and similar accuracy as the teacher model, while using minimal communication among students. Finally, as a case study, we deploy the proposed model for CIFAR-10 dataset on edge devices and demonstrate significant improvements in memory footprint (up to 24x), performance (up to 12x), and energy per node (up to 14x) compared to the large teacher model. We further show that for distributed inference on multiple edge devices, our proposed NoNN model results in up to 33x reduction in total latency w.r.t. a state-of-the-art model compression baseline.