# Communication-efficient Decentralized Machine Learning over Heterogeneous Networks

arXiv: https://arxiv.org/abs/2009.05766 · HTML: https://mlsanity.com/paper/2009.05766/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2020.

## Built on by (most cited first, 9 of 9)

- [Decentralized Federated Learning: A Survey and Perspective](https://mlsanity.com/paper/2306.01603/) (2023)
- [Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/2404.06114/) (2024)
- [Exploring the Impact of Serverless Computing on Peer To Peer Training Machine Learning](https://mlsanity.com/paper/2309.14139/) (2023)
- [p2pGNN: A Decentralized Graph Neural Network for Node Classification in Peer-to-Peer Networks](https://mlsanity.com/paper/2111.14837/) (2021)
- [Heterogeneity-Aware Cooperative Federated Edge Learning with Adaptive Computation and Communication Compression](https://mlsanity.com/paper/2409.04022/) (2024)
- [Towards Adapting Federated & Quantum Machine Learning for Network Intrusion Detection: A Survey](https://mlsanity.com/paper/2509.21389/) (2025)
- [The Metaverse Data Deluge: What Can We Do About It?](https://mlsanity.com/paper/2206.10326/) (2022)
- [Go With The Flow: Churn-Tolerant Decentralized Training of Large Language Models](https://mlsanity.com/paper/2509.21221/) (2025)
- [Towards Energy Impact on AI-Powered 6G IoT Networks: Centralized vs. Decentralized](https://mlsanity.com/paper/2604.19377/) (2026)

## Abstract (arXiv)

In the last few years, distributed machine learning has been usually executed over heterogeneous networks such as a local area network within a multi-tenant cluster or a wide area network connecting data centers and edge clusters. In these heterogeneous networks, the link speeds among worker nodes vary significantly, making it challenging for state-of-the-art machine learning approaches to perform efficient training. Both centralized and decentralized training approaches suffer from low-speed links. In this paper, we propose a decentralized approach, namely NetMax, that enables worker nodes to communicate via high-speed links and, thus, significantly speed up the training process. NetMax possesses the following novel features. First, it consists of a novel consensus algorithm that allows worker nodes to train model copies on their local dataset asynchronously and exchange information via peer-to-peer communication to synchronize their local copies, instead of a central master node (i.e., parameter server). Second, each worker node selects one peer randomly with a fine-tuned probability to exchange information per iteration. In particular, peers with high-speed links are selected with high probability. Third, the probabilities of selecting peers are designed to minimize the total convergence time. Moreover, we mathematically prove the convergence of NetMax. We evaluate NetMax on heterogeneous cluster networks and show that it achieves speedups of 3.7X, 3.4X, and 1.9X in comparison with the state-of-the-art decentralized training approaches Prague, Allreduce-SGD, and AD-PSGD, respectively.