Enhanced Federated Optimization: Adaptive Unbiased Client Sampling with Reduced Variance
Not on the map. Why?
Abstract (arXiv)
Federated Learning (FL) is a distributed learning paradigm to train a global model across multiple devices without collecting local data. In FL, a server typically selects a subset of clients for each training round to optimize resource usage. Central to this process is the technique of unbiased client sampling, which ensures a representative selection of clients. Current methods primarily utilize a random sampling procedure which, despite its effectiveness, achieves suboptimal efficiency owing to the loose upper bound caused by the sampling variance. In this work, by adopting an independent sampling procedure, we propose a federated optimization framework focused on adaptive unbiased client sampling, improving the convergence rate via an online variance reduction strategy. In particular, we present the first adaptive client sampler, K-Vib, employing an independent sampling procedure. K-Vib achieves a linear speed-up on the regret bound tildeObig(N^frac13T^frac23/K^frac43big) within a set communication budget K. Empirical studies indicate that K-Vib doubles the speed compared to baseline algorithms, demonstrating significant potential in federated optimization.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 52 of its 74 references that are in the map. First posted on arXiv 4 Oct 2023.
Builds on
52 of 74 references.
References · 52
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2021Convex optimizationcited by 2,479
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
- 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
- 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
- 2020Adaptive Federated Optimizationcited by 1,036
- 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
- 2018LEAF: A Benchmark for Federated Settingscited by 780
38 more
- 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
- 2021Federated Learning Based on Dynamic Regularizationcited by 555
- 2020An Efficient Framework for Clustered Federated Learningcited by 505
- 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
- 2019Fair Resource Allocation in Federated Learningcited by 389
- 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
- 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
- 2021A Field Guide to Federated Optimizationcited by 312
- 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 196
- 2020Federated Learning for Internet of Things: Recent Advances, Taxonomy, and Open Challengescited by 138
- 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 132
- 2020Optimal Client Sampling for Federated Learningcited by 123
- 2020Tilted Empirical Risk Minimizationcited by 103
- 2021Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learningcited by 93
- 2014Stochastic Optimization with Importance Samplingcited by 91
- 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
- 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 62
- 2021FedLab: A Flexible Federated Learning Frameworkcited by 54
- 2020Federated learning with class imbalance reductioncited by 52
- 2022Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learningcited by 33
- 2022On the Convergence of Clustered Federated Learningcited by 31
- 2023On the Convergence of Federated Averaging with Cyclic Client Participationcited by 31
- 2017Stochastic Primal-Dual Hybrid Gradient Algorithm with Arbitrary Sampling and Imaging Applicationscited by 29
- 2020Bandit-based Communication-Efficient Client Selection Strategies for Federated Learningcited by 29
- 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
- 2022DELTA: Diverse Client Sampling for Fasting Federated Learningcited by 24
- 2023Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initializationcited by 21
- 2013On Optimal Probabilities in Stochastic Coordinate Descent Methodscited by 14
- 2014Coordinate Descent with Arbitrary Sampling I: Algorithms and Complexitycited by 14
- 2018Nonconvex Variance Reduced Optimization with Arbitrary Samplingcited by 14
- 2023Federated Learning with Regularized Client Participationcited by 13
- 2018Online Variance Reduction for Stochastic Optimizationcited by 10
- 2021Adaptive Importance Sampling for Finite-Sum Optimization and Sampling with Decreasing Step-Sizescited by 9
- 2017Stochastic Optimization with Bandit Samplingcited by 8
- 2022Fast Heterogeneous Federated Learning with Hybrid Client Selectioncited by 8
- 2019Online Variance Reduction with Mixturescited by 7
- 2020Accurate and Fast Federated Learning via Combinatorial Multi-Armed Banditscited by 7
- 2022A Practical Cross-Device Federated Learning Framework over 5G Networkscited by 7
50 most cited of 52 All 52 (JSON)
Built on by
Citing papers per year: 2024: 1.
1 paper.