# Efficient and Reliable Overlay Networks for Decentralized Federated Learning

arXiv: https://arxiv.org/abs/2112.15486 · HTML: https://mlsanity.com/paper/2112.15486/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 59 references that are in the map.
- First posted on arXiv 12 Dec 2021.

## Builds on (most cited first, 30 of 30)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Train faster, generalize better: Stability of stochastic gradient descent](https://mlsanity.com/paper/1509.01240/) (2015)
- [Federated Learning with Differential Privacy: Algorithms and Performance Analysis](https://mlsanity.com/paper/1911.00222/) (2019)
- [Exploiting Unintended Feature Leakage in Collaborative Learning](https://mlsanity.com/paper/1805.04049/) (2018)
- [Differentially Private Federated Learning: A Client Level Perspective](https://mlsanity.com/paper/1712.07557/) (2017)
- [A Hybrid Approach to Privacy-Preserving Federated Learning](https://mlsanity.com/paper/1812.03224/) (2018)
- [On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization](https://mlsanity.com/paper/1905.03817/) (2019)
- [Practical Secure Aggregation for Federated Learning on User-Held Data](https://mlsanity.com/paper/1611.04482/) (2016)
- [FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data](https://mlsanity.com/paper/2005.11418/) (2020)
- [LDP-Fed: Federated Learning with Local Differential Privacy](https://mlsanity.com/paper/2006.03637/) (2020)
- [Decentralized Federated Averaging](https://mlsanity.com/paper/2104.11375/) (2021)
- [FedSplit: An algorithmic framework for fast federated optimization](https://mlsanity.com/paper/2005.05238/) (2020)
- [Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms](https://mlsanity.com/paper/1808.07576/) (2018)
- [Deep learning with Elastic Averaging SGD](https://mlsanity.com/paper/1412.6651/) (2014)
- [MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling](https://mlsanity.com/paper/1905.09435/) (2019)
- [Privacy For Free: Wireless Federated Learning Via Uncoded Transmission With Adaptive Power Control](https://mlsanity.com/paper/2006.05459/) (2020)
- [Federated Learning with Bayesian Differential Privacy](https://mlsanity.com/paper/1911.10071/) (2019)
- [HybridAlpha: An Efficient Approach for Privacy-Preserving Federated Learning](https://mlsanity.com/paper/1912.05897/) (2019)
- [Throughput-Optimal Topology Design for Cross-Silo Federated Learning](https://mlsanity.com/paper/2010.12229/) (2020)
- [FedSel: Federated SGD under Local Differential Privacy with Top-k Dimension Selection](https://mlsanity.com/paper/2003.10637/) (2020)
- [Gradient-Leaks: Understanding and Controlling Deanonymization in Federated Learning](https://mlsanity.com/paper/1805.05838/) (2018)
- [Anonymizing Data for Privacy-Preserving Federated Learning](https://mlsanity.com/paper/2002.09096/) (2020)
- [Expander Graph and Communication-Efficient Decentralized Optimization](https://mlsanity.com/paper/1612.00903/) (2016)
- [A Distributed Trust Framework for Privacy-Preserving Machine Learning](https://mlsanity.com/paper/2006.02456/) (2020)
- [Fast Distributed Coordinate Descent for Non-Strongly Convex Losses](https://mlsanity.com/paper/1405.5300/) (2014)
- [Enhancing the Privacy of Federated Learning with Sketching](https://mlsanity.com/paper/1911.01812/) (2019)

## Built on by (most cited first, 9 of 9)

- [Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges](https://mlsanity.com/paper/2211.08413/) (2022)
- [Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learning](https://mlsanity.com/paper/2310.03461/) (2023)
- [Review of Mathematical Optimization in Federated Learning](https://mlsanity.com/paper/2412.01630/) (2024)
- [Decentralized Multi-Level Compositional Optimization Algorithms with Level-Independent Convergence Rate](https://mlsanity.com/paper/2306.03322/) (2023)
- [Exposing the Vulnerability of Decentralized Learning to Membership Inference Attacks Through the Lens of Graph Mixing](https://mlsanity.com/paper/2412.12837/) (2024)
- [Secure Decentralized Learning with Blockchain](https://mlsanity.com/paper/2310.07079/) (2023)
- [Heterogeneity-Aware Resource Allocation and Topology Design for Hierarchical Federated Edge Learning](https://mlsanity.com/paper/2409.19509/) (2024)
- [Learning Decentralized Swarms Using Rotation Equivariant Graph Neural Networks](https://mlsanity.com/paper/2502.17612/) (2025)
- [Communication Optimization for Decentralized Learning atop Bandwidth-limited Edge Networks](https://mlsanity.com/paper/2504.12210/) (2025)

## Abstract (arXiv)

We propose near-optimal overlay networks based on d-regular expander graphs to accelerate decentralized federated learning (DFL) and improve its generalization. In DFL a massive number of clients are connected by an overlay network, and they solve machine learning problems collaboratively without sharing raw data. Our overlay network design integrates spectral graph theory and the theoretical convergence and generalization bounds for DFL. As such, our proposed overlay networks accelerate convergence, improve generalization, and enhance robustness to clients failures in DFL with theoretical guarantees. Also, we present an efficient algorithm to convert a given graph to a practical overlay network and maintaining the network topology after potential client failures. We numerically verify the advantages of DFL with our proposed networks on various benchmark tasks, ranging from image classification to language modeling using hundreds of clients.