# Decentralized Federated Learning: Balancing Communication and Computing Costs

arXiv: https://arxiv.org/abs/2107.12048 · HTML: https://mlsanity.com/paper/2107.12048/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 26 Jul 2021.

## Builds on (most cited first, 26 of 26)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Local SGD Converges Fast and Communicates Little](https://mlsanity.com/paper/1805.09767/) (2018)
- [Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/1705.09056/) (2017)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [Adaptive Federated Learning in Resource Constrained Edge Computing Systems](https://mlsanity.com/paper/1804.05271/) (2018)
- [Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning](https://mlsanity.com/paper/1807.06629/) (2018)
- [Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication](https://mlsanity.com/paper/1902.00340/) (2019)
- [On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization](https://mlsanity.com/paper/1905.03817/) (2019)
- [On the Convergence of Decentralized Gradient Descent](https://mlsanity.com/paper/1310.7063/) (2013)
- [Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization](https://mlsanity.com/paper/1506.08272/) (2015)
- [Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms](https://mlsanity.com/paper/1808.07576/) (2018)
- [Decentralized Collaborative Learning of Personalized Models over Networks](https://mlsanity.com/paper/1610.05202/) (2016)
- [Accelerating Federated Learning via Momentum Gradient Descent](https://mlsanity.com/paper/1910.03197/) (2019)
- [Decentralized Federated Learning: A Segmented Gossip Approach](https://mlsanity.com/paper/1908.07782/) (2019)
- [Collaborative Deep Learning in Fixed Topology Networks](https://mlsanity.com/paper/1706.07880/) (2017)
- [ByRDiE: Byzantine-resilient distributed coordinate descent for decentralized learning](https://mlsanity.com/paper/1708.08155/) (2017)
- [Robust Federated Learning with Noisy Communication](https://mlsanity.com/paper/1911.00251/) (2019)
- [Decentralized Consensus Algorithm with Delayed and Stochastic Gradients](https://mlsanity.com/paper/1604.05649/) (2016)
- [Communication-Censored Linearized ADMM for Decentralized Consensus Optimization](https://mlsanity.com/paper/1909.06724/) (2019)

## Built on by (most cited first, 30 of 55)

- [Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges](https://mlsanity.com/paper/2211.08413/) (2022)
- [Green Edge AI: A Contemporary Survey](https://mlsanity.com/paper/2312.00333/) (2023)
- [Decentralized Federated Learning: A Survey on Security and Privacy](https://mlsanity.com/paper/2401.17319/) (2024)
- [Topology-aware Federated Learning in Edge Computing: A Comprehensive Survey](https://mlsanity.com/paper/2302.02573/) (2023)
- [Unleashing the Power of Continual Learning on Non-Centralized Devices: A Survey](https://mlsanity.com/paper/2412.13840/) (2024)
- [An Energy and Carbon Footprint Analysis of Distributed and Federated Learning](https://mlsanity.com/paper/2206.10380/) (2022)
- [AEDFL: Efficient Asynchronous Decentralized Federated Learning with Heterogeneous Devices](https://mlsanity.com/paper/2312.10935/) (2023)
- [Fedstellar: A Platform for Decentralized Federated Learning](https://mlsanity.com/paper/2306.09750/) (2023)
- [Semi-Decentralized Federated Edge Learning with Data and Device Heterogeneity](https://mlsanity.com/paper/2112.10313/) (2021)
- [Advances in Robust Federated Learning: A Survey with Heterogeneity Considerations](https://mlsanity.com/paper/2405.09839/) (2024)
- [Confederated Learning: Federated Learning with Decentralized Edge Servers](https://mlsanity.com/paper/2205.14905/) (2022)
- [Advancements in Federated Learning: Models, Methods, and Privacy](https://mlsanity.com/paper/2302.11466/) (2023)
- [Communication-Efficient Design for Quantized Decentralized Federated Learning](https://mlsanity.com/paper/2303.08423/) (2023)
- [Distributed Learning over Networks with Graph-Attention-Based Personalization](https://mlsanity.com/paper/2305.13041/) (2023)
- [Stochastic Unrolled Federated Learning](https://mlsanity.com/paper/2305.15371/) (2023)
- [Blockchained Federated Learning for Internet of Things: A Comprehensive Survey](https://mlsanity.com/paper/2305.04513/) (2023)
- [Jamming Attacks on Decentralized Federated Learning in General Multi-Hop Wireless Networks](https://mlsanity.com/paper/2301.05250/) (2023)
- [On the Computation-Communication Trade-Off with A Flexible Gradient Tracking Approach](https://mlsanity.com/paper/2306.07159/) (2023)
- [Generalization Error Matters in Decentralized Learning Under Byzantine Attacks](https://mlsanity.com/paper/2407.08632/) (2024)
- [NTK-DFL: Enhancing Decentralized Federated Learning in Heterogeneous Settings via Neural Tangent Kernel](https://mlsanity.com/paper/2410.01922/) (2024)
- [A Carbon Tracking Model for Federated Learning: Impact of Quantization and Sparsification](https://mlsanity.com/paper/2310.08087/) (2023)
- [Distributed Event-Based Learning via ADMM](https://mlsanity.com/paper/2405.10618/) (2024)
- [Federated and Transfer Learning for Cancer Detection Based on Image Analysis](https://mlsanity.com/paper/2405.20126/) (2024)
- [FedUHB: Accelerating Federated Unlearning via Polyak Heavy Ball Method](https://mlsanity.com/paper/2411.11039/) (2024)
- [Communication Compression for Decentralized Learning with Operator Splitting Methods](https://mlsanity.com/paper/2205.03779/) (2022)
- [Sparse Decentralized Federated Learning](https://mlsanity.com/paper/2308.16671/) (2023)
- [Topology Learning for Heterogeneous Decentralized Federated Learning over Unreliable D2D Networks](https://mlsanity.com/paper/2312.13611/) (2023)
- [ProFe: Communication-Efficient Decentralized Federated Learning via Distillation and Prototypes](https://mlsanity.com/paper/2412.11207/) (2024)
- [Privatized Graph Federated Learning](https://mlsanity.com/paper/2203.07105/) (2022)
- [Enhanced Decentralized Federated Learning based on Consensus in Connected Vehicles](https://mlsanity.com/paper/2209.10722/) (2022)

## Abstract (arXiv)

Decentralized stochastic gradient descent (SGD) is a driving engine for decentralized federated learning (DFL). The performance of decentralized SGD is jointly influenced by inter-node communications and local updates. In this paper, we propose a general DFL framework, which implements both multiple local updates and multiple inter-node communications periodically, to strike a balance between communication efficiency and model consensus. It can provide a general decentralized SGD analytical framework. We establish strong convergence guarantees for the proposed DFL algorithm without the assumption of convex objectives. The convergence rate of DFL can be optimized to achieve the balance of communication and computing costs under constrained resources. For improving communication efficiency of DFL, compressed communication is further introduced to the proposed DFL as a new scheme, named DFL with compressed communication (C-DFL). The proposed C-DFL exhibits linear convergence for strongly convex objectives. Experiment results based on MNIST and CIFAR-10 datasets illustrate the superiority of DFL over traditional decentralized SGD methods and show that C-DFL further enhances communication efficiency.