mlsanityData to 7 Oct 2026

Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Sampling

arXiv 2112.11256 ↗First posted 21 Dec 2021cs.LG cs.AI cs.DCcited by 66

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) algorithms usually sample a fraction of clients in each round (partial participation) when the number of participants is large and the server's communication bandwidth is limited. Recent works on the convergence analysis of FL have focused on unbiased client sampling, e.g., sampling uniformly at random, which suffers from slow wall-clock time for convergence due to high degrees of system heterogeneity and statistical heterogeneity. This paper aims to design an adaptive client sampling algorithm that tackles both system and statistical heterogeneity to minimize the wall-clock convergence time. We obtain a new tractable convergence bound for FL algorithms with arbitrary client sampling probabilities. Based on the bound, we analytically establish the relationship between the total learning time and sampling probabilities, which results in a non-convex optimization problem for training time minimization. We design an efficient algorithm for learning the unknown parameters in the convergence bound and develop a low-complexity algorithm to approximately solve the non-convex problem. Experimental results from both hardware prototype and simulation demonstrate that our proposed sampling scheme significantly reduces the convergence time compared to several baseline sampling schemes. Notably, our scheme in hardware prototype spends 73% less time than the uniform sampling baseline for reaching the same target loss.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 37 of its 47 references that are in the map. First posted on arXiv 21 Dec 2021.

Builds on

37 of 47 references.

References · 37

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2021Convex optimizationcited by 2,479
  5. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  6. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  7. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  8. 2019Towards Federated Learning at Scale: System Designcited by 826
  9. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  10. 2019Federated Machine Learning: Concept and Applicationscited by 409
  11. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 408
  12. 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 347
25 more
  1. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  2. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 196
  3. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  4. 2016Practical Secure Aggregation for Federated Learning on User-Held Datacited by 178
  5. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 178
  6. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  7. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 150
  8. 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 132
  9. 2019Scheduling Policies for Federated Learning in Wireless Networkscited by 131
  10. 2020Optimal Client Sampling for Federated Learningcited by 123
  11. 2020TiFL: A Tier-based Federated Learning Systemcited by 95
  12. 2014Stochastic Optimization with Importance Samplingcited by 91
  13. 2020Joint Device Scheduling and Resource Allocation for Latency Constrained Wireless Federated Learningcited by 90
  14. 2020Convergence Time Optimization for Federated Learning over Wireless Networkscited by 81
  15. 2015Variance Reduction in SGD by Distributed Importance Samplingcited by 79
  16. 2018Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGDcited by 76
  17. 2020Fast-Convergent Federated Learningcited by 61
  18. 2020Cost-Effective Federated Learning Designcited by 61
  19. 2020Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneitycited by 56
  20. 2021Device Sampling for Heterogeneous Federated Learning: Theory, Algorithms, and Implementationcited by 40
  21. 2020Network-Aware Optimization of Distributed Learning for Fog Computingcited by 27
  22. 2020Federated Learning under Importance Samplingcited by 26
  23. 2021Cost-Effective Federated Learning in Mobile Edge Networkscited by 25
  24. 2017BLENDER: Enabling Local Search with a Hybrid Differential Privacy Modelcited by 14
  25. 2021Faithful Edge Federated Learning: Scalability and Privacycited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 16; 2024: 26; 2025: 11; 2026: 3.

66 papers.

Sort:

By area:

  1. 2022Client Selection in Federated Learning: Principles, Challenges, and Opportunitiescited by 62
  2. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
  3. 2022DELTA: Diverse Client Sampling for Fasting Federated Learningcited by 24
  4. 2023Over-The-Air Federated Learning: Status Quo, Open Challenges, and Future Directionscited by 21
  5. 2022Scheduling and Aggregation Design for Asynchronous Federated Learning over Wireless Networkscited by 20
  6. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  7. 2022GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexitycited by 17
  8. 2023Data and Model Poisoning Backdoor Attacks on Wireless Federated Learning, and the Defense Mechanisms: A Comprehensive Surveycited by 17
  9. 2023DYNAMITE: Dynamic Interplay of Mini-Batch Size and Aggregation Frequency for Federated Learning with Static and Streaming Datasetcited by 14
  10. 2023Federated Learning with Regularized Client Participationcited by 13
  11. 2023MergeSFL: Split Federated Learning with Feature Merging and Batch Size Regulationcited by 13
  12. 2022Federated Learning with Flexible Controlcited by 12
38 more
  1. 2024Green Federated Learning: A new era of Green Aware AIcited by 12
  2. 2022A Survey on Participant Selection for Federated Learning in Mobile Networkscited by 11
  3. 2023Federated Learning Across Decentralized and Unshared Archives for Remote Sensing Image Classificationcited by 10
  4. 2024Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selectioncited by 10
  5. 2024Adaptive Heterogeneous Client Sampling for Federated Learning over Wireless Networkscited by 9
  6. 2023Incentive Mechanism Design for Unbiased Federated Learning with Randomized Client Participationcited by 8
  7. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 8
  8. 2022GlueFL: Reconciling Client Sampling and Model Masking for Bandwidth Efficient Federated Learningcited by 7
  9. 2023Optimization Design for Federated Learning in Heterogeneous 6G Networkscited by 7
  10. 2022Flow: Per-Instance Personalized Federated Learning Through Dynamic Routingcited by 6
  11. 2023Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networkscited by 5
  12. 2024Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devicescited by 5
  13. 2025FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learningcited by 4
  14. 2022Adaptive Control of Client Selection and Gradient Compression for Efficient Federated Learningcited by 3
  15. 2023Stabilizing and Improving Federated Learning with Non-IID Data and Client Dropoutcited by 3
  16. 2024FedAST: Federated Asynchronous Simultaneous Trainingcited by 3
  17. 2024Review of Mathematical Optimization in Federated Learningcited by 3
  18. 2023Collaborative Mean Estimation over Intermittently Connected Networks with Peer-To-Peer Privacycited by 2
  19. 2023FedDD: Toward Communication-efficient Federated Learning with Differential Parameter Dropoutcited by 2
  20. 2023Straggler-resilient Federated Learning: Tackling Computation Heterogeneity with Layer-wise Partial Model Training in Mobile Edge Networkcited by 2
  21. 2024Maverick-Aware Shapley Valuation for Client Selection in Federated Learningcited by 2
  22. 2024Privacy Preserving Semi-Decentralized Mean Estimation over Intermittently-Connected Networkscited by 2
  23. 2024Addressing Heterogeneity in Federated Learning: Challenges and Solutions for a Shared Production Environmentcited by 2
  24. 2025FedFetch: Faster Federated Learning with Adaptive Downstream Prefetchingcited by 2
  25. 2023LEFL: Low Entropy Client Sampling in Federated Learningcited by 1
  26. 2024Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codescited by 1
  27. 2024Federated Learning Can Find Friends That Are Advantageouscited by 1
  28. 2024Adaptive Federated Learning in Heterogeneous Wireless Networks with Independent Samplingcited by 1
  29. 2024FLARE: A New Federated Learning Framework with Adjustable Learning Rates over Resource-Constrained Wireless Networkscited by 1
  30. 2024Sequential Federated Learning in Hierarchical Architecture on Non-IID Datasetscited by 1
  31. 2025Incentivizing Multi-Tenant Split Federated Learning for Foundation Models at the Network Edgecited by 1
  32. 2025Client Selection in Federated Learning with Data Heterogeneity and Network Latenciescited by 1
  33. 2025A Federated Random Forest Solution for Secure Distributed Machine Learningcited by 1
  34. 2025Heterogeneity-Aware Client Sampling for Optimal and Efficient Federated Learningcited by 1
  35. 2025Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Samplingcited by 1
  36. 2022Depersonalized Federated Learning: Tackling Statistical Heterogeneity by Alternating Stochastic Gradient Descentcited by 0
  37. 2023FedAgg: Adaptive Federated Learning with Aggregated Gradientscited by 0
  38. 2024GPFL: A Gradient Projection-Based Client Selection Framework for Efficient Federated Learningcited by 0

50 most cited of 66 All 66 (JSON)