mlsanityData to 14 Oct 2026

Heterogeneity-Guided Client Sampling: Towards Fast and Efficient Non-IID Federated Learning

arXiv 2310.00198 ↗First posted 30 Sep 2023cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Statistical heterogeneity of data present at client devices in a federated learning (FL) system renders the training of a global model in such systems difficult. Particularly challenging are the settings where due to communication resource constraints only a small fraction of clients can participate in any given round of FL. Recent approaches to training a global model in FL systems with non-IID data have focused on developing client selection methods that aim to sample clients with more informative updates of the model. However, existing client selection techniques either introduce significant computation overhead or perform well only in the scenarios where clients have data with similar heterogeneity profiles. In this paper, we propose HiCS-FL (Federated Learning via Hierarchical Clustered Sampling), a novel client selection method in which the server estimates statistical heterogeneity of a client's data using the client's update of the network's output layer and relies on this information to cluster and sample the clients. We analyze the ability of the proposed techniques to compare heterogeneity of different datasets, and characterize convergence of the training process that deploys the introduced client selection method. Extensive experimental results demonstrate that in non-IID settings HiCS-FL achieves faster convergence than state-of-the-art FL client selection schemes. Notably, HiCS-FL drastically reduces computation cost compared to existing selection schemes and is adaptable to different heterogeneity scenarios.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 30 Sep 2023.

Builds on

36 of 49 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  7. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  8. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  9. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  10. 2021Model-Contrastive Federated Learningcited by 766
  11. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  12. 2020Personalized Federated Learning with Moreau Envelopescited by 618
24 more
  1. 2020Federated Learning with Matched Averagingcited by 564
  2. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  3. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  4. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 513
  5. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  6. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  7. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  8. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 198
  9. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  10. 2016Recurrent Neural Network for Text Classification with Multi-Task Learningcited by 144
  11. 2020Optimal Client Sampling for Federated Learningcited by 124
  12. 2015Relay Backpropagation for Effective Learning of Deep Convolutional Neural Networkscited by 115
  13. 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
  14. 2021Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learningcited by 93
  15. 2021Towards Fair Federated Learning with Zero-Shot Data Augmentationcited by 64
  16. 2019Active Federated Learningcited by 63
  17. 2020Federated learning with class imbalance reductioncited by 52
  18. 2021FedCor: Correlation-Based Active Client Selection Strategy for Heterogeneous Federated Learningcited by 43
  19. 2020Communication-Efficient Federated Learning via Optimal Client Samplingcited by 39
  20. 2022Fed-CBS: A Heterogeneity-Aware Client Sampling Mechanism for Federated Learning via Class-Imbalance Reductioncited by 32
  21. 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillationcited by 25
  22. 2020Budgeted Online Selection of Candidate IoT Clients to Participate in Federated Learningcited by 14
  23. 2022Federated Learning in Non-IID Settings Aided by Differentially Private Synthetic Datacited by 12
  24. 2020Accurate and Fast Federated Learning via Combinatorial Multi-Armed Banditscited by 7

Built on by

Cited by year

Citing papers per year: 2023: 2.

2 papers.

  1. 2023Mixed-Precision Quantization for Federated Learning on Resource-Constrained Heterogeneous Devicescited by 11
  2. 2023Enhanced Federated Optimization: Adaptive Unbiased Client Sampling with Reduced Variancecited by 2