# Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication

arXiv: https://arxiv.org/abs/1805.08768 · HTML: https://mlsanity.com/paper/1805.08768/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2018.

## Built on by (most cited first, 30 of 51)

- [Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications](https://mlsanity.com/paper/2207.09353/) (2022)
- [FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings](https://mlsanity.com/paper/2210.04620/) (2022)
- [Decentralized Federated Learning: A Survey and Perspective](https://mlsanity.com/paper/2306.01603/) (2023)
- [Communication Efficiency in Federated Learning: Achievements and Challenges](https://mlsanity.com/paper/2107.10996/) (2021)
- [Differentially Private Federated Learning with Local Regularization and Sparsification](https://mlsanity.com/paper/2203.03106/) (2022)
- [Knowledge Distillation for Federated Learning: a Practical Guide](https://mlsanity.com/paper/2211.04742/) (2022)
- [FEDZIP: A Compression Framework for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2102.01593/) (2021)
- [On the Utility of Gradient Compression in Distributed Training Systems](https://mlsanity.com/paper/2103.00543/) (2021)
- [Reducing Communication for Split Learning by Randomized Top-k Sparsification](https://mlsanity.com/paper/2305.18469/) (2023)
- [Fast Federated Learning by Balancing Communication Trade-Offs](https://mlsanity.com/paper/2105.11028/) (2021)
- [ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization](https://mlsanity.com/paper/2311.13171/) (2023)
- [Efficient and Secure Federated Learning for Financial Applications](https://mlsanity.com/paper/2303.08355/) (2023)
- [1-Bit Compressive Sensing for Efficient Federated Learning Over the Air](https://mlsanity.com/paper/2103.16055/) (2021)
- [Communication and Computation Reduction for Split Learning using Asynchronous Training](https://mlsanity.com/paper/2107.09786/) (2021)
- [Temporal-Structure-Assisted Gradient Aggregation for Over-the-Air Federated Edge Learning](https://mlsanity.com/paper/2103.02270/) (2021)
- [Compressed Particle-Based Federated Bayesian Learning and Unlearning](https://mlsanity.com/paper/2209.07267/) (2022)
- [A Privacy Preserving System for Movie Recommendations Using Federated Learning](https://mlsanity.com/paper/2303.04689/) (2023)
- [EvoFed: Leveraging Evolutionary Strategies for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2311.07485/) (2023)
- [Communication-Efficient Federated Learning via Predictive Coding](https://mlsanity.com/paper/2108.00918/) (2021)
- [Data-Aware Gradient Compression for FL in Communication-Constrained Mobile Computing](https://mlsanity.com/paper/2311.07324/) (2023)
- [Accelerating Federated Edge Learning via Topology Optimization](https://mlsanity.com/paper/2204.00489/) (2022)
- [Client Selection for Federated Bayesian Learning](https://mlsanity.com/paper/2212.05492/) (2022)
- [ResFed: Communication Efficient Federated Learning by Transmitting Deep Compressed Residuals](https://mlsanity.com/paper/2212.05602/) (2022)
- [Federated Neural Radiance Fields](https://mlsanity.com/paper/2305.01163/) (2023)
- [Distributed Learning based on 1-Bit Gradient Coding in the Presence of Stragglers](https://mlsanity.com/paper/2403.14716/) (2024)
- [SRATTA : Sample Re-ATTribution Attack of Secure Aggregation in Federated Learning](https://mlsanity.com/paper/2306.07644/) (2023)
- [FedQV: Leveraging Quadratic Voting in Federated Learning](https://mlsanity.com/paper/2401.01168/) (2024)
- [Adaptive Scheduling for Machine Learning Tasks over Networks](https://mlsanity.com/paper/2101.10007/) (2021)
- [Linear Regression over Networks with Communication Guarantees](https://mlsanity.com/paper/2103.04140/) (2021)
- [Reward-Based 1-bit Compressed Federated Distillation on Blockchain](https://mlsanity.com/paper/2106.14265/) (2021)

## Abstract (arXiv)

Currently, progressively larger deep neural networks are trained on ever growing data corpora. As this trend is only going to increase in the future, distributed training schemes are becoming increasingly relevant. A major issue in distributed training is the limited communication bandwidth between contributing nodes or prohibitive communication cost in general. These challenges become even more pressing, as the number of computation nodes increases. To counteract this development we propose sparse binary compression (SBC), a compression framework that allows for a drastic reduction of communication cost for distributed training. SBC combines existing techniques of communication delay and gradient sparsification with a novel binarization method and optimal weight update encoding to push compression gains to new limits. By doing so, our method also allows us to smoothly trade-off gradient sparsity and temporal sparsity to adapt to the requirements of the learning task. Our experiments show, that SBC can reduce the upstream communication on a variety of convolutional and recurrent neural network architectures by more than four orders of magnitude without significantly harming the convergence speed in terms of forward-backward passes. For instance, we can train ResNet50 on ImageNet in the same number of iterations to the baseline accuracy, using × 3531 less bits or train it to a 1% lower accuracy using × 37208 less bits. In the latter case, the total upstream communication required is cut from 125 terabytes to 3.35 gigabytes for every participating client.