# Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection

arXiv: https://arxiv.org/abs/2002.09692 · HTML: https://mlsanity.com/paper/2002.09692/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2020.

## Built on by (most cited first, 20 of 20)

- [Decentralized Federated Learning: A Survey and Perspective](https://mlsanity.com/paper/2306.01603/) (2023)
- [Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matching](https://mlsanity.com/paper/2203.12285/) (2022)
- [Decentralized Federated Learning for UAV Networks: Architecture, Challenges, and Opportunities](https://mlsanity.com/paper/2104.07557/) (2021)
- [FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs](https://mlsanity.com/paper/2309.01172/) (2023)
- [FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion](https://mlsanity.com/paper/2410.20380/) (2024)
- [Mobility-Aware Federated Self-supervised Learning in Vehicular Network](https://mlsanity.com/paper/2408.00256/) (2024)
- [Get More for Less in Decentralized Learning Systems](https://mlsanity.com/paper/2306.04377/) (2023)
- [Boosting Asynchronous Decentralized Learning with Model Fragmentation](https://mlsanity.com/paper/2410.12918/) (2024)
- [Efficient Ring-topology Decentralized Federated Learning with Deep Generative Models for Industrial Artificial Intelligent](https://mlsanity.com/paper/2104.08100/) (2021)
- [A review of Federated Learning in Intrusion Detection Systems for IoT](https://mlsanity.com/paper/2204.12443/) (2022)
- [Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2408.14736/) (2024)
- [Exploring the Impact of Serverless Computing on Peer To Peer Training Machine Learning](https://mlsanity.com/paper/2309.14139/) (2023)
- [Protocol Learning, Decentralized Frontier Risk and the No-Off Problem](https://mlsanity.com/paper/2412.07890/) (2024)
- [DeceFL: A Principled Decentralized Federated Learning Framework](https://mlsanity.com/paper/2107.07171/) (2021)
- [Communication-Efficient Secure Aggregation in Decentralized Learning](https://mlsanity.com/paper/2405.07708/) (2024)
- [Decentralized Entropic Optimal Transport for Distributed Distribution Comparison](https://mlsanity.com/paper/2301.12065/) (2023)
- [FedSAUC: A Similarity-Aware Update Control for Communication-Efficient Federated Learning in Edge Computing](https://mlsanity.com/paper/2504.04867/) (2025)
- [Mosaic Learning: A Framework for Decentralized Learning with Model Fragmentation](https://mlsanity.com/paper/2602.04352/) (2026)
- [MAR-FL: A Communication Efficient Peer-to-Peer Federated Learning System](https://mlsanity.com/paper/2512.05234/) (2025)
- [Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization](https://mlsanity.com/paper/2605.23464/) (2026)

## Abstract (arXiv)

Distributed learning techniques such as federated learning have enabled multiple workers to train machine learning models together to reduce the overall training time. However, current distributed training algorithms (centralized or decentralized) suffer from the communication bottleneck on multiple low-bandwidth workers (also on the server under the centralized architecture). Although decentralized algorithms generally have lower communication complexity than the centralized counterpart, they still suffer from the communication bottleneck for workers with low network bandwidth. To deal with the communication problem while being able to preserve the convergence performance, we introduce a novel decentralized training algorithm with the following key features: 1) It does not require a parameter server to maintain the model during training, which avoids the communication pressure on any single peer. 2) Each worker only needs to communicate with a single peer at each communication round with a highly compressed model, which can significantly reduce the communication traffic on the worker. We theoretically prove that our sparsification algorithm still preserves convergence properties. 3) Each worker dynamically selects its peer at different communication rounds to better utilize the bandwidth resources. We conduct experiments with convolutional neural networks on 32 workers to verify the effectiveness of our proposed algorithm compared to seven existing methods. Experimental results show that our algorithm significantly reduces the communication traffic and generally select relatively high bandwidth peers.