mlsanityData to 7 Oct 2026

CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning

arXiv 2106.07155 ↗First posted 14 Jun 2021cs.LG cs.DCcited by 6

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is a prevailing distributed learning paradigm, where a large number of workers jointly learn a model without sharing their training data. However, high communication costs could arise in FL due to large-scale (deep) learning models and bandwidth-constrained connections. In this paper, we introduce a communication-efficient algorithmic framework called CFedAvg for FL with non-i.i.d. datasets, which works with general (biased or unbiased) SNR-constrained compressors. We analyze the convergence rate of CFedAvg for non-convex functions with constant and decaying learning rates. The CFedAvg algorithm can achieve an O(1 / √mKT + 1 / T) convergence rate with a constant learning rate, implying a linear speedup for convergence as the number of workers increases, where K is the number of local steps, T is the number of total communication rounds, and m is the total worker number. This matches the convergence rate of distributed/federated learning without compression, thus achieving high communication efficiency while not sacrificing learning accuracy in FL. Furthermore, we extend CFedAvg to cases with heterogeneous local steps, which allows different workers to perform a different number of local steps to better adapt to their own circumstances. The interesting observation in general is that the noise/variance introduced by compressors does not affect the overall convergence rate order for non-i.i.d. FL. We verify the effectiveness of our CFedAvg algorithm on three datasets with two gradient compression schemes of different compression ratios.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 34 of its 44 references that are in the map. First posted on arXiv 14 Jun 2021.

Builds on

34 of 44 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2019Advances and Open Problems in Federated Learningcited by 3,021
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  7. 2018Federated Learning with Non-IID Datacited by 1,099
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  10. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  11. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  12. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 415
22 more
  1. 2018Sparsified SGD with Memorycited by 373
  2. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  3. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  4. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
  5. 2017Sparse Communication for Distributed Gradient Descentcited by 243
  6. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  7. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  8. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 199
  9. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  10. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 179
  11. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 178
  12. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  13. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  14. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  15. 2019Decentralized Deep Learning with Arbitrary Communication Compressioncited by 109
  16. 2018Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithmscited by 106
  17. 2017On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimizationcited by 91
  18. 2019textttDeepSqueeze: Decentralization Meets Error-Compensated Compressioncited by 21
  19. 2020Private and Communication-Efficient Edge Learning: A Sparse Differential Gaussian-Masking Distributed SGD Approachcited by 18
  20. 2020Dynamic Federated Learningcited by 12
  21. 2018LoAdaBoost: loss-based AdaBoost federated machine learning with reduced computational complexity on IID and non-IID intensive care datacited by 9
  22. 2019Communication-Efficient Network-Distributed Optimization with Differential-Coded Compressorscited by 3

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 3; 2025: 2.

6 papers.

  1. 2022NET-FLEET: Achieving Linear Convergence Speedup for Fully Decentralized Federated Learning with Heterogeneous Datacited by 12
  2. 2024Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimizationcited by 12
  3. 2025Converge Faster, Talk Less: Hessian-Informed Federated Zeroth-Order Optimizationcited by 3
  4. 2024Expediting In-Network Federated Learning by Voting-Based Consensus Model Compressioncited by 2
  5. 2024Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codescited by 1
  6. 2025γ-FedHT: Stepsize-Aware Hard-Threshold Gradient Compression in Federated Learningcited by 1