# Demystifying Why Local Aggregation Helps: Convergence Analysis of Hierarchical SGD

arXiv: https://arxiv.org/abs/2010.12998 · HTML: https://mlsanity.com/paper/2010.12998/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2020.

## Built on by (most cited first, 26 of 26)

- [Mobility-Aware Cluster Federated Learning in Hierarchical Wireless Networks](https://mlsanity.com/paper/2108.09103/) (2021)
- [Scalable and Low-Latency Federated Learning with Cooperative Mobile Edge Networking](https://mlsanity.com/paper/2205.13054/) (2022)
- [Mobility Accelerates Learning: Convergence Analysis on Hierarchical Federated Learning in Vehicular Networks](https://mlsanity.com/paper/2401.09656/) (2024)
- [Multi-Edge Server-Assisted Dynamic Federated Learning with an Optimized Floating Aggregation Point](https://mlsanity.com/paper/2203.13950/) (2022)
- [Adaptive Federated Pruning in Hierarchical Wireless Networks](https://mlsanity.com/paper/2305.09042/) (2023)
- [Hierarchical Federated Learning in Wireless Networks: Pruning Tackles Bandwidth Scarcity and System Heterogeneity](https://mlsanity.com/paper/2308.01562/) (2023)
- [Satellite Federated Edge Learning: Architecture Design and Convergence Analysis](https://mlsanity.com/paper/2404.01875/) (2024)
- [Delay Sensitive Hierarchical Federated Learning with Stochastic Local Updates](https://mlsanity.com/paper/2302.04851/) (2023)
- [Confederated Learning: Federated Learning with Decentralized Edge Servers](https://mlsanity.com/paper/2205.14905/) (2022)
- [Flame: Simplifying Topology Extension in Federated Learning](https://mlsanity.com/paper/2305.05118/) (2023)
- [Hierarchical Federated Learning with Multi-Timescale Gradient Correction](https://mlsanity.com/paper/2409.18448/) (2024)
- [FedFog: Network-Aware Optimization of Federated Learning over Wireless Fog-Cloud Systems](https://mlsanity.com/paper/2107.02755/) (2021)
- [Hierarchical Federated Learning with Momentum Acceleration in Multi-Tier Networks](https://mlsanity.com/paper/2210.14560/) (2022)
- [Data-Aware Gradient Compression for FL in Communication-Constrained Mobile Computing](https://mlsanity.com/paper/2311.07324/) (2023)
- [Resource-Aware Hierarchical Federated Learning in Wireless Video Caching Networks](https://mlsanity.com/paper/2402.04216/) (2024)
- [Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods](https://mlsanity.com/paper/2406.13936/) (2024)
- [WW-FL: Secure and Private Large-Scale Federated Learning](https://mlsanity.com/paper/2302.09904/) (2023)
- [UAV-assisted Unbiased Hierarchical Federated Learning: Performance and Convergence Analysis](https://mlsanity.com/paper/2407.07739/) (2024)
- [Hierarchical Over-the-Air FedGradNorm](https://mlsanity.com/paper/2212.07414/) (2022)
- [Federated Learning over Hierarchical Wireless Networks: Training Latency Minimization via Submodel Partitioning](https://mlsanity.com/paper/2310.17890/) (2023)
- [Resource-Aware Hierarchical Federated Learning for Video Caching in Wireless Networks](https://mlsanity.com/paper/2311.06918/) (2023)
- [A Hierarchical Federated Learning Approach for the Internet of Things](https://mlsanity.com/paper/2403.01540/) (2024)
- [Personalized Multi-tier Federated Learning](https://mlsanity.com/paper/2407.14251/) (2024)
- [Heterogeneity-Aware Resource Allocation and Topology Design for Hierarchical Federated Edge Learning](https://mlsanity.com/paper/2409.19509/) (2024)
- [Personalized Hierarchical Split Federated Learning in Wireless Networks](https://mlsanity.com/paper/2411.06042/) (2024)
- [Fed-BAC: Federated Bandit-Guided Additive Clustering in Hierarchical Federated Learning](https://mlsanity.com/paper/2605.11815/) (2026)

## Abstract (arXiv)

Hierarchical SGD (H-SGD) has emerged as a new distributed SGD algorithm for multi-level communication networks. In H-SGD, before each global aggregation, workers send their updated local models to local servers for aggregations. Despite recent research efforts, the effect of local aggregation on global convergence still lacks theoretical understanding. In this work, we first introduce a new notion of "upward" and "downward" divergences. We then use it to conduct a novel analysis to obtain a worst-case convergence upper bound for two-level H-SGD with non-IID data, non-convex objective function, and stochastic gradient. By extending this result to the case with random grouping, we observe that this convergence upper bound of H-SGD is between the upper bounds of two single-level local SGD settings, with the number of local iterations equal to the local and global update periods in H-SGD, respectively. We refer to this as the "sandwich behavior". Furthermore, we extend our analytical approach based on "upward" and "downward" divergences to study the convergence for the general case of H-SGD with more than two levels, where the "sandwich behavior" still holds. Our theoretical results provide key insights of why local aggregation can be beneficial in improving the convergence of H-SGD.