# Multi-Level Local SGD for Heterogeneous Hierarchical Networks

arXiv: https://arxiv.org/abs/2007.13819 · HTML: https://mlsanity.com/paper/2007.13819/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jul 2020.

## Built on by (most cited first, 9 of 9)

- [Mobility-Aware Cluster Federated Learning in Hierarchical Wireless Networks](https://mlsanity.com/paper/2108.09103/) (2021)
- [Semi-Decentralized Federated Edge Learning for Fast Convergence on Non-IID Data](https://mlsanity.com/paper/2104.12678/) (2021)
- [Semi-Decentralized Federated Edge Learning with Data and Device Heterogeneity](https://mlsanity.com/paper/2112.10313/) (2021)
- [Multi-Tier Federated Learning for Vertically Partitioned Data](https://mlsanity.com/paper/2102.03620/) (2021)
- [Cross-Silo Federated Learning for Multi-Tier Networks with Vertical and Horizontal Data Partitioning](https://mlsanity.com/paper/2108.08930/) (2021)
- [Hierarchical Federated Learning with Momentum Acceleration in Multi-Tier Networks](https://mlsanity.com/paper/2210.14560/) (2022)
- [Robust Federated Learning with Connectivity Failures: A Semi-Decentralized Framework with Collaborative Relaying](https://mlsanity.com/paper/2202.11850/) (2022)
- [Multi-Model Federated Learning with Provable Guarantees](https://mlsanity.com/paper/2207.04330/) (2022)
- [Asynchronous Semi-Decentralized Federated Edge Learning for Heterogeneous Clients](https://mlsanity.com/paper/2112.04737/) (2021)

## Abstract (arXiv)

We propose Multi-Level Local SGD, a distributed gradient method for learning a smooth, non-convex objective in a heterogeneous multi-level network. Our network model consists of a set of disjoint sub-networks, with a single hub and multiple worker nodes; further, worker nodes may have different operating rates. The hubs exchange information with one another via a connected, but not necessarily complete communication network. In our algorithm, sub-networks execute a distributed SGD algorithm, using a hub-and-spoke paradigm, and the hubs periodically average their models with neighboring hubs. We first provide a unified mathematical framework that describes the Multi-Level Local SGD algorithm. We then present a theoretical analysis of the algorithm; our analysis shows the dependence of the convergence error on the worker node heterogeneity, hub network topology, and the number of local, sub-network, and global iterations. We back up our theoretical results via simulation-based experiments using both convex and non-convex objectives.