A Lightweight Method for Tackling Unknown Participation Statistics in Federated Averaging
Not on the map. Why?
Abstract (arXiv)
In federated learning (FL), clients usually have diverse participation statistics that are unknown a priori, which can significantly harm the performance of FL if not handled properly. Existing works aiming at addressing this problem are usually based on global variance reduction, which requires a substantial amount of additional memory in a multiplicative factor equal to the total number of clients. An important open problem is to find a lightweight method for FL in the presence of clients with unknown participation rates. In this paper, we address this problem by adapting the aggregation weights in federated averaging (FedAvg) based on the participation history of each client. We first show that, with heterogeneous participation statistics, FedAvg with non-optimal aggregation weights can diverge from the optimal solution of the original FL objective, indicating the need of finding optimal aggregation weights. However, it is difficult to compute the optimal weights when the participation statistics are unknown. To address this problem, we present a new algorithm called FedAU, which improves FedAvg by adaptively weighting the client updates based on online estimates of the optimal weights without knowing the statistics of client participation. We provide a theoretical convergence analysis of FedAU using a novel methodology to connect the estimation error and convergence. Our theoretical results reveal important and interesting insights, while showing that FedAU converges to an optimal solution of the original objective and has desirable properties such as linear speedup. Our experimental results also verify the advantage of FedAU over baseline methods with various participation patterns.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 36 of its 53 references that are in the map. First posted on arXiv 6 Jun 2023.
Builds on
36 of 53 references.
References · 36
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
- 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
- 2020Adaptive Federated Optimizationcited by 1,036
- 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
- 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
- 2019Towards Federated Learning at Scale: System Designcited by 826
- 2018Local SGD Converges Fast and Communicates Littlecited by 595
- 2021A Field Guide to Federated Optimizationcited by 312
- 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
24 more
- 2019Differentially Private Learning with Adaptive Clippingcited by 218
- 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
- 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
- 2020Optimal Client Sampling for Federated Learningcited by 123
- 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 106
- 2021Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learningcited by 93
- 2020Local SGD: Unified Theory and New Efficient Methodscited by 88
- 2018Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGDcited by 76
- 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
- 2020Fast-Convergent Federated Learning with Adaptive Weightingcited by 67
- 2020Towards Flexible Device Participation in Federated Learningcited by 64
- 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 62
- 2022Communication-Efficient Adaptive Federated Learningcited by 55
- 2022A Unified Analysis of Federated Learning with Arbitrary Client Participationcited by 45
- 2021Anarchic Federated Learningcited by 42
- 2019Semi-Cyclic Stochastic Gradient Descentcited by 40
- 2023On the Convergence of Federated Averaging with Cyclic Client Participationcited by 31
- 2022Communication-Efficient Device Scheduling for Federated Learning Using Stochastic Optimizationcited by 28
- 2020Federated Learning under Importance Samplingcited by 26
- 2020Distributed Non-Convex Optimization with Sublinear Speedup under Intermittent Client Availabilitycited by 20
- 2022AsyncFedED: Asynchronous Federated Learning with Euclidean Distance based Adaptive Weight Aggregationcited by 19
- 2023Federated Learning with Regularized Client Participationcited by 13
- 2021Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimizationcited by 12
- 2020Distributed Optimization over Block-Cyclic Datacited by 6
Built on by
Citing papers per year: 2024: 1.
1 paper.