mlsanityData to 7 Oct 2026

SCAFFOLD: Stochastic Controlled Averaging for Federated Learning

arXiv 1910.06378 ↗First posted 14 Oct 2019cs.LG cs.DC math.OCcited by 174

Not on the map. Why?

Abstract (arXiv)

Federated Averaging (FedAvg) has emerged as the algorithm of choice for federated learning due to its simplicity and low communication cost. However, in spite of recent research efforts, its performance is not fully understood. We obtain tight convergence rates for FedAvg and prove that it suffers from `client-drift' when the data is heterogeneous (non-iid), resulting in unstable and slow convergence. As a solution, we propose a new algorithm (SCAFFOLD) which uses control variates (variance reduction) to correct for the `client-drift' in its local updates. We prove that SCAFFOLD requires significantly fewer communication rounds and is not affected by data heterogeneity or client sampling. Further, we show that (for quadratics) SCAFFOLD can take advantage of similarity in the client's data yielding even faster convergence. The latter is the first result to quantify the usefulness of local-steps in distributed optimization.

Key facts. Cited by 174 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 42; 2022: 52; 2023: 31; 2024: 20; 2025: 21; 2026: 8.

174 papers.

Sort:

By area:

  1. 2021Model-Contrastive Federated Learningcited by 762
  2. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  3. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  4. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  5. 2021Personalized Federated Learning using Hypernetworkscited by 210
  6. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  7. 2021Distributionally Robust Federated Averagingcited by 104
  8. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  9. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  10. 2021FedProc: Prototypical Contrastive Federated Learning on Non-IID datacited by 80
  11. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 76
  12. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
38 more
  1. 2022An Improved Analysis of Gradient Tracking for Decentralized Machine Learningcited by 68
  2. 2021Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Samplingcited by 66
  3. 2021HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Imagescited by 62
  4. 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 62
  5. 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
  6. 2021FedNL: Making Newton-Type Methods Applicable to Federated Learningcited by 50
  7. 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillationcited by 45
  8. 2022pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learningcited by 44
  9. 2021FedCor: Correlation-Based Active Client Selection Strategy for Heterogeneous Federated Learningcited by 43
  10. 2021Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiencycited by 40
  11. 2021FedCG: Leverage Conditional GAN for Protecting Privacy and Maintaining Competitive Performance in Federated Learningcited by 39
  12. 2021Scaling Federated Learning for Fine-tuning of Large Language Modelscited by 36
  13. 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Datacited by 36
  14. 2021Device-Cloud Collaborative Learning for Recommendationcited by 34
  15. 2022Multi-Level Branched Regularization for Federated Learningcited by 34
  16. 2022Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learningcited by 33
  17. 2021Cross-Gradient Aggregation for Decentralized Learning from Non-IID datacited by 32
  18. 2021FedJAX: Federated learning simulation with JAXcited by 32
  19. 2022Fed-CBS: A Heterogeneity-Aware Client Sampling Mechanism for Federated Learning via Class-Imbalance Reductioncited by 32
  20. 2024Relaxed Contrastive Learning for Federated Learningcited by 32
  21. 2021Local Adaptivity in Federated Learning: Convergence and Consistencycited by 31
  22. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Datacited by 31
  23. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 31
  24. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  25. 2023On the Convergence of Federated Averaging with Cyclic Client Participationcited by 31
  26. 2023One-Shot Federated Conformal Predictioncited by 26
  27. 2022Communication-Efficient Federated Learning with Accelerated Client Gradientcited by 24
  28. 2022DYNAFED: Tackling Client Data Heterogeneity with Global Dynamicscited by 22
  29. 2022Can 5th Generation Local Training Methods Support Client Sampling? Yes!cited by 22
  30. 2021Federated Deep AUC Maximization for Heterogeneous Data with a Constant Communication Complexitycited by 21
  31. 2022Local Stochastic Bilevel Optimization with Momentum-Based Variance Reductioncited by 20
  32. 2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 19
  33. 2022Communication Acceleration of Local Gradient Methods via an Accelerated Primal-Dual Algorithm with Inexact Proxcited by 19
  34. 2023Achieving Linear Speedup in Non-IID Federated Bilevel Learningcited by 18
  35. 2022GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexitycited by 17
  36. 2021Differential Privacy Meets Federated Learning under Communication Constraintscited by 15
  37. 2021Distributed Methods with Compressed Communication for Solving Variational Inequalities, with Theoretical Guaranteescited by 15
  38. 2021FLoRA: Single-shot Hyper-parameter Optimization for Federated Learningcited by 14

50 most cited of 174 All 174 (JSON)