# Decentralized Learning Made Easy with DecentralizePy

arXiv: https://arxiv.org/abs/2304.08322 · HTML: https://mlsanity.com/paper/2304.08322/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Builds on (most cited first, 23 of 23)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/1705.09056/) (2017)
- [Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training](https://mlsanity.com/paper/1712.01887/) (2017)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [A Unified Theory of Decentralized SGD with Changing Topology and Local Updates](https://mlsanity.com/paper/2003.10422/) (2020)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [FedML: A Research Library and Benchmark for Federated Machine Learning](https://mlsanity.com/paper/2007.13518/) (2020)
- [The Convergence of Sparsified Gradient Methods](https://mlsanity.com/paper/1809.10505/) (2018)
- [Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication](https://mlsanity.com/paper/1902.00340/) (2019)
- [Applied Federated Learning: Improving Google Keyboard Query Suggestions](https://mlsanity.com/paper/1812.02903/) (2018)
- [FedScale: Benchmarking Model and System Performance of Federated Learning at Scale](https://mlsanity.com/paper/2105.11367/) (2021)
- [Decentralized Deep Learning with Arbitrary Communication Compression](https://mlsanity.com/paper/1907.09356/) (2019)
- [NVIDIA FLARE: Federated Learning from Simulation to Real-World](https://mlsanity.com/paper/2210.13291/) (2022)
- [IPFS - Content Addressed, Versioned, P2P File System](https://mlsanity.com/paper/1407.3561/) (2014)
- [Topology-aware Generalization of Decentralized SGD](https://mlsanity.com/paper/2206.12680/) (2022)
- [Beyond spectral gap: The role of the topology in decentralized learning](https://mlsanity.com/paper/2206.03093/) (2022)
- [IPLS : A Framework for Decentralized Federated Learning](https://mlsanity.com/paper/2101.01901/) (2021)
- [TEE-based decentralized recommender systems: The raw data sharing redemption](https://mlsanity.com/paper/2202.11655/) (2022)
- [Kollaps: Decentralized and Dynamic Topology Emulation](https://mlsanity.com/paper/2004.02253/) (2020)

## Built on by (most cited first, 18 of 18)

- [Epidemic Learning: Boosting Decentralized Learning with Randomized Communication](https://mlsanity.com/paper/2310.01972/) (2023)
- [Noiseless Privacy-Preserving Decentralized Learning](https://mlsanity.com/paper/2404.09536/) (2024)
- [Get More for Less in Decentralized Learning Systems](https://mlsanity.com/paper/2306.04377/) (2023)
- [Boosting Asynchronous Decentralized Learning with Model Fragmentation](https://mlsanity.com/paper/2410.12918/) (2024)
- [Low-Cost Privacy-Preserving Decentralized Learning](https://mlsanity.com/paper/2403.11795/) (2024)
- [Analyzing Inference Privacy Risks Through Gradients in Machine Learning](https://mlsanity.com/paper/2408.16913/) (2024)
- [Communication-Efficient Secure Aggregation in Decentralized Learning](https://mlsanity.com/paper/2405.07708/) (2024)
- [Dynamic Topology Optimization for Non-IID Data in Decentralized Learning](https://mlsanity.com/paper/2602.03383/) (2026)
- [Energy-Aware Decentralized Learning with Intermittent Model Training](https://mlsanity.com/paper/2407.01283/) (2024)
- [Fair Decentralized Learning](https://mlsanity.com/paper/2410.02541/) (2024)
- [Heterogeneous Decentralized Diffusion Models](https://mlsanity.com/paper/2603.06741/) (2026)
- [Your Neighbors Know: Leveraging Local Neighborhoods for Backdoor Detection in Decentralized Learning](https://mlsanity.com/paper/2605.19969/) (2026)
- [Harnessing Increased Client Participation with Cohort-Parallel Federated Learning](https://mlsanity.com/paper/2405.15644/) (2024)
- [On the Robustness of Distributed Machine Learning against Transfer Attacks](https://mlsanity.com/paper/2412.14080/) (2024)
- [Accelerating MoE Model Inference with Expert Sharding](https://mlsanity.com/paper/2503.08467/) (2025)
- [Hubs and Spokes Learning: Efficient and Scalable Collaborative Machine Learning](https://mlsanity.com/paper/2504.20988/) (2025)
- [Efficient Pyramidal Analysis of Gigapixel Images on a Decentralized Modest Computer Cluster](https://mlsanity.com/paper/2509.02440/) (2025)
- [Decentralized SGD with Controlled Disagreement Finds Flatter Minima](https://mlsanity.com/paper/2602.02899/) (2026)

## Abstract (arXiv)

Decentralized learning (DL) has gained prominence for its potential benefits in terms of scalability, privacy, and fault tolerance. It consists of many nodes that coordinate without a central server and exchange millions of parameters in the inherently iterative process of machine learning (ML) training. In addition, these nodes are connected in complex and potentially dynamic topologies. Assessing the intricate dynamics of such networks is clearly not an easy task. Often in literature, researchers resort to simulated environments that do not scale and fail to capture practical and crucial behaviors, including the ones associated to parallelism, data transfer, network delays, and wall-clock time. In this paper, we propose DecentralizePy, a distributed framework for decentralized ML, which allows for the emulation of large-scale learning networks in arbitrary topologies. We demonstrate the capabilities of DecentralizePy by deploying techniques such as sparsification and secure aggregation on top of several topologies, including dynamic networks with more than one thousand nodes.