mlsanityData to 14 Oct 2026

Fast Federated Learning by Balancing Communication Trade-Offs

arXiv 2105.11028 ↗First posted 23 May 2021cs.LG cs.AI cs.DCcited by 18

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) has recently received a lot of attention for large-scale privacy-preserving machine learning. However, high communication overheads due to frequent gradient transmissions decelerate FL. To mitigate the communication overheads, two main techniques have been studied: (i) local update of weights characterizing the trade-off between communication and computation and (ii) gradient compression characterizing the trade-off between communication and precision. To the best of our knowledge, studying and balancing those two trade-offs jointly and dynamically while considering their impacts on convergence has remained unresolved even though it promises significantly faster FL. In this paper, we first formulate our problem to minimize learning error with respect to two variables: local update coefficients and sparsity budgets of gradient compression who characterize trade-offs between communication and computation/precision, respectively. We then derive an upper bound of the learning error in a given wall-clock time considering the interdependency between the two variables. Based on this theoretical analysis, we propose an enhanced FL scheme, namely Fast FL (FFL), that jointly and dynamically adjusts the two variables to minimize the learning error. We demonstrate that FFL consistently achieves higher accuracies faster than similar schemes existing in the literature.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 23 May 2021.

Builds on

19 of 26 references.

References · 19

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  5. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  6. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  7. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  8. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 409
  9. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  10. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  11. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  12. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 136
7 more
  1. 2019Convergence of Edge Computing and Deep Learning: A Comprehensive Surveycited by 120
  2. 2018Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithmscited by 106
  3. 2017On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimizationcited by 92
  4. 2018Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGDcited by 77
  5. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  6. 2018Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communicationcited by 51
  7. 2013Petuum: A New Platform for Distributed Machine Learning on Big Datacited by 35

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 2; 2023: 7; 2024: 6; 2025: 1.

18 papers.

Sort:

By area:

  1. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 187
  2. 2022Trading Off Privacy, Utility and Efficiency in Federated Learningcited by 33
  3. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  4. 2024Distributed Deep Reinforcement Learning Based Gradient Quantization for Federated Learning Enabled Vehicle Edge Computingcited by 15
  5. 2023Optimizing Privacy, Utility and Efficiency in Constrained Multi-Objective Federated Learningcited by 12
  6. 2022MOB-FL: Mobility-Aware Federated Learning for Intelligent Connected Vehiclescited by 8
  7. 2023Towards Achieving Near-optimal Utility for Privacy-Preserving Federated Learning via Data Generation and Parameter Distortioncited by 6
  8. 2023Theoretically Principled Federated Learning for Balancing Privacy and Utilitycited by 6
  9. 2023A Meta-learning Framework for Tuning Parameters of Protection Mechanisms in Trustworthy Federated Learningcited by 6
  10. 2024Communication-Efficient Hybrid Federated Learning for E-health with Horizontal and Vertical Data Partitioningcited by 6
  11. 2024Quantized and Asynchronous Federated Learningcited by 3
  12. 2021Federated learning and next generation wireless communications: A survey on bidirectional relationshipcited by 2
6 more
  1. 2024Heterogeneity-Aware Cooperative Federated Edge Learning with Adaptive Computation and Communication Compressioncited by 2
  2. 2025Communication-Efficient Federated Learning by Quantized Variance Reduction for Heterogeneous Wireless Edge Networkscited by 2
  3. 2023GQFedWAvg: Optimization-Based Quantized Federated Learning in General Edge Computing Systemscited by 1
  4. 2024A Joint Approach to Local Updating and Gradient Compression for Efficient Asynchronous Federated Learningcited by 1
  5. 2023REWAFL: Residual Energy and Wireless Aware Participant Selection for Efficient Federated Learning over Mobile Devicescited by 0
  6. 2024H-FedSN: Personalized Sparse Networks for Efficient and Accurate Hierarchical Federated Learning for IoT Applicationscited by 0