mlsanityData to 7 Oct 2026

Accelerating Federated Learning by Selecting Beneficial Herd of Local Gradients

arXiv 2403.16557 ↗First posted 25 Mar 2024cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) is a distributed machine learning framework in communication network systems. However, the systems' Non-Independent and Identically Distributed (Non-IID) data negatively affect the convergence efficiency of the global model, since only a subset of these data samples are beneficial for model convergence. In pursuit of this subset, a reliable approach involves determining a measure of validity to rank the samples within the dataset. In this paper, We propose the BHerd strategy which selects a beneficial herd of local gradients to accelerate the convergence of the FL model. Specifically, we map the distribution of the local dataset to the local gradients and use the Herding strategy to obtain a permutation of the set of gradients, where the more advanced gradients in the permutation are closer to the average of the set of gradients. These top portion of the gradients will be selected and sent to the server for global aggregation. We conduct experiments on different datasets, models and scenarios by building a prototype system, and experimental results demonstrate that our BHerd strategy is effective in selecting beneficial local gradients to mitigate the effects brought by the Non-IID dataset, thus accelerating model convergence.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 59 references that are in the map. First posted on arXiv 25 Mar 2024.

Builds on

38 of 59 references.

References · 38

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  5. 2018Federated Learning with Non-IID Datacited by 1,099
  6. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  7. 2021Domain Generalization: A Surveycited by 760
  8. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  9. 2017Federated Multi-Task Learningcited by 669
  10. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  11. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  12. 2021Federated Learning Based on Dynamic Regularizationcited by 555
26 more
  1. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  2. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  3. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 415
  4. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 408
  5. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  6. 2021Federated Learning on Non-IID Data: A Surveycited by 345
  7. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  8. 2018Federated Meta-Learning with Fast Convergence and Efficient Communicationcited by 154
  9. 2019Federated Adversarial Domain Adaptationcited by 140
  10. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
  11. 2021Decentralized Federated Averagingcited by 114
  12. 2020Random Reshuffling: Simple Analysis with Vast Improvementscited by 112
  13. 2018Computational Optimal Transport: Complexity by Accelerated Gradient Descent Is Better Than by Sinkhorn's Algorithmcited by 111
  14. 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
  15. 2018Tight Analyses for Non-Smooth Stochastic Gradient Descentcited by 94
  16. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  17. 2019Variational Federated Multi-Task Learningcited by 69
  18. 2021From Distributed Machine Learning to Federated Learning: A Surveycited by 65
  19. 2020Federated learning with class imbalance reductioncited by 52
  20. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
  21. 2022FairGRAPE: Fairness-aware GRAdient Pruning mEthod for Face Attribute Classificationcited by 30
  22. 2022Mitigating Data Heterogeneity in Federated Learning with Data Augmentationcited by 21
  23. 2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 19
  24. 2022GraB: Finding Provably Better Data Permutations than Random Reshufflingcited by 18
  25. 2023TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphscited by 16
  26. 2022AI-based Fog and Edge Computing: A Systematic Review, Taxonomy and Future Directionscited by 4

Built on by

Not cited yet.