mlsanityData to 14 Oct 2026

Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies

arXiv 2109.03999 ↗First posted 9 Sep 2021cs.DC cs.LG cs.NIcited by 9

Not on the map. Why?

Abstract (arXiv)

The increasing demand for privacy-preserving collaborative learning has given rise to a new computing paradigm called federated learning (FL), in which clients collaboratively train a machine learning (ML) model without revealing their private training data. Given an acceptable level of privacy guarantee, the goal of FL is to minimize the time-to-accuracy of model training. Compared with distributed ML in data centers, there are four distinct challenges to achieving short time-to-accuracy in FL training, namely the lack of information for optimization, the tradeoff between statistical and system utility, client heterogeneity, and large configuration space. In this paper, we survey recent works in addressing these challenges and present them following a typical training workflow through three phases: client selection, configuration, and reporting. We also review system works including measurement studies and benchmarking tools that aim to support FL developers.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 101 of its 175 references that are in the map. First posted on arXiv 9 Sep 2021.

Builds on

101 of 175 references.

References · 101

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  6. 2019Advances and Open Problems in Federated Learningcited by 3,046
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  8. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  9. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  10. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  11. 2014How transferable are features in deep neural networks?cited by 1,225
  12. 2020Adaptive Federated Optimizationcited by 1,046
38 more
  1. 2019Deep Leakage from Gradientscited by 1,021
  2. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  3. 2019Towards Federated Learning at Scale: System Designcited by 832
  4. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  5. 2018LEAF: A Benchmark for Federated Settingscited by 792
  6. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 711
  7. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  8. 2017Federated Multi-Task Learningcited by 672
  9. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  10. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 580
  11. 2018Federated Learning for Mobile Keyboard Predictioncited by 572
  12. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 520
  13. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  14. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  15. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 493
  16. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  17. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  18. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
  19. 2019Federated Machine Learning: Concept and Applicationscited by 412
  20. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 408
  21. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  22. 2018Sparsified SGD with Memorycited by 376
  23. 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 352
  24. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  25. 2018The Hidden Vulnerability of Distributed Learning in Byzantiumcited by 329
  26. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  27. 2021A Field Guide to Federated Optimizationcited by 315
  28. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  29. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  30. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 300
  31. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  32. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  33. 2017Sparse Communication for Distributed Gradient Descentcited by 249
  34. 2018Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learningcited by 246
  35. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 204
  36. 2019Client-Edge-Cloud Hierarchical Federated Learningcited by 202
  37. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
  38. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180

50 most cited of 101 All 101 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 2; 2025: 2; 2026: 1.

9 papers.

  1. 2022Pisces: Efficient Federated Learning via Guided Asynchronous Trainingcited by 8
  2. 2025SyncFed: Time-Aware Federated Learning through Explicit Timestamping and Synchronizationcited by 4
  3. 2022Quantifying the Impact of Label Noise on Federated Learningcited by 3
  4. 2024Adaptive Active Inference Agents for Heterogeneous and Lifelong Federated Learningcited by 3
  5. 2024Review of Mathematical Optimization in Federated Learningcited by 3
  6. 2025Federated Learning Survey: A Multi-Level Taxonomy of Aggregation Techniques, Experimental Insights, and Future Frontierscited by 3
  7. 2023A Safe Deep Reinforcement Learning Approach for Energy Efficient Federated Learning in Wireless Communication Networkscited by 2
  8. 2023Federated Learning in IoT: a Survey from a Resource-Constrained Perspectivecited by 2
  9. 2026On the Push-Based Asynchronous Federated Learning: A Bias-Correction Aggregation Approachcited by 0