# DISCO: Distributed Inference with Sparse Communications

arXiv: https://arxiv.org/abs/2302.11180 · HTML: https://mlsanity.com/paper/2302.11180/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 57 references that are in the map.
- First posted on arXiv 22 Feb 2023.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Sparse Communication for Distributed Gradient Descent](https://mlsanity.com/paper/1704.05021/) (2017)
- [Distributed Deep Neural Networks over the Cloud, the Edge and End Devices](https://mlsanity.com/paper/1709.01921/) (2017)
- [PipeDream: Fast and Efficient Pipeline Parallel DNN Training](https://mlsanity.com/paper/1806.03377/) (2018)
- [JointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Services](https://mlsanity.com/paper/1801.08618/) (2018)
- [Supporting Very Large Models using Automatic Dataflow Graph Partitioning](https://mlsanity.com/paper/1807.08887/) (2018)
- [Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform](https://mlsanity.com/paper/1809.02839/) (2018)
- [Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks](https://mlsanity.com/paper/1802.04924/) (2018)
- [XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training](https://mlsanity.com/paper/1911.04610/) (2019)
- [Learned Gradient Compression for Distributed Deep Learning](https://mlsanity.com/paper/2103.08870/) (2021)
- [SplitNets: Designing Neural Architectures for Efficient Distributed Computing on Head-Mounted Systems](https://mlsanity.com/paper/2204.04705/) (2022)
- [Latency and Throughput Characterization of Convolutional Neural Networks for Mobile Computer Vision](https://mlsanity.com/paper/1803.09492/) (2018)
- [AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks](https://mlsanity.com/paper/1705.09786/) (2017)
- [DistrEdge: Speeding up Convolutional Neural Network Inference on Distributed Edge Devices](https://mlsanity.com/paper/2202.01699/) (2022)
- [Automatic Graph Partitioning for Very Large-scale Deep Learning](https://mlsanity.com/paper/2103.16063/) (2021)
- [Nested Dithered Quantization for Communication Reduction in Distributed Training](https://mlsanity.com/paper/1904.01197/) (2019)
- [Optimization Framework for Splitting DNN Inference Jobs over Computing Networks](https://mlsanity.com/paper/2111.07006/) (2021)
- [DEFER: Distributed Edge Inference for Deep Neural Networks](https://mlsanity.com/paper/2201.06769/) (2022)

## Built on by (most cited first, 2 of 2)

- [Towards Low-bit Communication for Tensor Parallel LLM Inference](https://mlsanity.com/paper/2411.07942/) (2024)
- [Going Beyond the Edge: Distributed Inference of Transformer Models on Ultra-Low-Power Wireless Devices](https://mlsanity.com/paper/2605.15694/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) have great potential to solve many real-world problems, but they usually require an extensive amount of computation and memory. It is of great difficulty to deploy a large DNN model to a single resource-limited device with small memory capacity. Distributed computing is a common approach to reduce single-node memory consumption and to accelerate the inference of DNN models. In this paper, we explore the "within-layer model parallelism", which distributes the inference of each layer into multiple nodes. In this way, the memory requirement can be distributed to many nodes, making it possible to use several edge devices to infer a large DNN model. Due to the dependency within each layer, data communications between nodes during this parallel inference can be a bottleneck when the communication bandwidth is limited. We propose a framework to train DNN models for Distributed Inference with Sparse Communications (DISCO). We convert the problem of selecting which subset of data to transmit between nodes into a model optimization problem, and derive models with both computation and communication reduction when each layer is inferred on multiple nodes. We show the benefit of the DISCO framework on a variety of CV tasks such as image classification, object detection, semantic segmentation, and image super resolution. The corresponding models include important DNN building blocks such as convolutions and transformers. For example, each layer of a ResNet-50 model can be distributively inferred across two nodes with five times less data communications, almost half overall computations and half memory requirement for a single node, and achieve comparable accuracy to the original ResNet-50 model. This also results in 4.7 times overall inference speedup.