mlsanityData to 14 Oct 2026

Boosting Resource-Constrained Federated Learning Systems with Guessed Updates

arXiv 2110.11486 ↗First posted 21 Oct 2021cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) enables a set of client devices to collaboratively train a model without sharing raw data. This process, though, operates under the constrained computation and communication resources of edge devices. These constraints combined with systems heterogeneity force some participating clients to perform fewer local updates than expected by the server, thus slowing down convergence. Exhaustive tuning of hyperparameters in FL, furthermore, can be resource-intensive, without which the convergence is adversely affected. In this work, we propose GEL, the guess and learn algorithm. GEL enables constrained edge devices to perform additional learning through guessed updates on top of gradient-based steps. These guesses are gradientless, i.e., participating clients leverage them for free. Our generic guessing algorithm (i) can be flexibly combined with several state-of-the-art algorithms including FEDPROX, FEDNOVA, FEDYOGI or SCALEFL; and (ii) achieves significantly improved performance when the learning rates are not best tuned. We conduct extensive experiments and show that GEL can boost empirical convergence by up to 40% in resource constrained networks while relieving the need for exhaustive learning rate tuning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 58 references that are in the map. First posted on arXiv 21 Oct 2021.

Builds on

33 of 58 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2019Advances and Open Problems in Federated Learningcited by 3,046
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  10. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  11. 2020Adaptive Federated Optimizationcited by 1,046
  12. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
21 more
  1. 2019Towards Federated Learning at Scale: System Designcited by 832
  2. 2018LEAF: A Benchmark for Federated Settingscited by 792
  3. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  4. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 536
  5. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  6. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  7. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  8. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 204
  9. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  10. 2016Practical Secure Aggregation for Federated Learning on User-Held Datacited by 178
  11. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 136
  12. 2018Tunability: Importance of Hyperparameters of Machine Learning Algorithmscited by 99
  13. 2020Acceleration for Compressed Gradient Descent in Distributed and Federated Optimizationcited by 90
  14. 2020Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learningcited by 78
  15. 2021On Large-Cohort Training for Federated Learningcited by 74
  16. 2021Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharingcited by 58
  17. 2020On the Outsized Importance of Learning Rates in Local Update Methodscited by 36
  18. 2020Importance of Tuning Hyperparameters of Machine Learning Algorithmscited by 20
  19. 2018Step Size Matters in Deep Learningcited by 19
  20. 2021A Review on Edge Analytics: Issues, Challenges, Opportunities, Promises, Future Directions, and Applicationscited by 5
  21. 2021AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruningcited by 4

Built on by

Not cited yet.