# Communication-efficient Distributed Cooperative Learning with Compressed Beliefs

arXiv: https://arxiv.org/abs/2102.07767 · HTML: https://mlsanity.com/paper/2102.07767/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 14 Feb 2021.

## Builds on (most cited first, 23 of 23)

- [Sparsified SGD with Memory](https://mlsanity.com/paper/1809.07599/) (2018)
- [Error Feedback Fixes SignSGD and other Gradient Compression Schemes](https://mlsanity.com/paper/1901.09847/) (2019)
- [Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication](https://mlsanity.com/paper/1902.00340/) (2019)
- [QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding](https://mlsanity.com/paper/1610.02132/) (2016)
- [On Biased Compression for Distributed Learning](https://mlsanity.com/paper/2002.12410/) (2020)
- [Decentralized Deep Learning with Arbitrary Communication Compression](https://mlsanity.com/paper/1907.09356/) (2019)
- [A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!](https://mlsanity.com/paper/2011.01697/) (2020)
- [Fast Convergence Rates for Distributed Non-Bayesian Learning](https://mlsanity.com/paper/1508.05161/) (2015)
- [Social Learning and Distributed Hypothesis Testing](https://mlsanity.com/paper/1410.4307/) (2014)
- [Social Learning over Weakly-Connected Graphs](https://mlsanity.com/paper/1609.03703/) (2016)
- [Nonasymptotic Convergence Rates for Cooperative Learning Over Time-Varying Directed Graphs](https://mlsanity.com/paper/1410.1977/) (2014)
- [Information Exchange and Learning Dynamics over Weakly-Connected Adaptive Networks](https://mlsanity.com/paper/1412.1523/) (2014)
- [Interplay between Topology and Social Learning over Weak Graphs](https://mlsanity.com/paper/1910.13905/) (2019)
- [Distributed Learning for Cooperative Inference](https://mlsanity.com/paper/1704.02718/) (2017)
- [Resilient Distributed Estimation: Sensor Attacks](https://mlsanity.com/paper/1709.06156/) (2017)
- [Graph-Theoretic Analysis of Belief System Dynamics under Logic Constraints](https://mlsanity.com/paper/1810.02456/) (2018)
- [Distributed Inference with Sparse and Quantized Communication](https://mlsanity.com/paper/2004.01302/) (2020)
- [Adaptation in Online Social Learning](https://mlsanity.com/paper/2003.01948/) (2020)
- [Non-Bayesian Social Learning with Uncertain Models over Time-Varying Directed Graphs](https://mlsanity.com/paper/1909.04255/) (2019)
- [Non-Bayesian Social Learning with Gaussian Uncertain Models](https://mlsanity.com/paper/1910.11251/) (2019)
- [Social Learning with Partial Information Sharing](https://mlsanity.com/paper/1910.13889/) (2019)
- [Distributed Hypothesis Testing and Social Learning in Finite Time with a Finite Amount of Communication](https://mlsanity.com/paper/2004.01306/) (2020)
- [Social learning under inferential attacks](https://mlsanity.com/paper/2010.13660/) (2020)

## Built on by (most cited first, 4 of 4)

- [On Generalization of Decentralized Learning with Separable Data](https://mlsanity.com/paper/2209.07116/) (2022)
- [Scalable Average Consensus with Compressed Communications](https://mlsanity.com/paper/2109.06996/) (2021)
- [On Arbitrary Compression for Decentralized Consensus and Stochastic Optimization over Directed Networks](https://mlsanity.com/paper/2204.08160/) (2022)
- [Achieving Linear Speedup with ProxSkip in Distributed Stochastic Optimization](https://mlsanity.com/paper/2310.07983/) (2023)

## Abstract (arXiv)

We study the problem of distributed cooperative learning, where a group of agents seeks to agree on a set of hypotheses that best describes a sequence of private observations. In the scenario where the set of hypotheses is large, we propose a belief update rule where agents share compressed (either sparse or quantized) beliefs with an arbitrary positive compression rate. Our algorithm leverages a unified communication rule that enables agents to access wide-ranging compression operators as black-box modules. We prove the almost sure asymptotic exponential convergence of beliefs around the set of optimal hypotheses. Additionally, we show a non-asymptotic, explicit, and linear concentration rate in probability of the beliefs on the optimal hypothesis set. We provide numerical experiments to illustrate the communication benefits of our method. The simulation results show that the number of transmitted bits can be reduced to 5-10% of the non-compressed method in the studied scenarios.