mlsanityData to 14 Oct 2026

Stochastic Controlled Averaging for Federated Learning with Communication Compression

arXiv 2308.08165 ↗First posted 16 Aug 2023math.OC cs.DC cs.LGcited by 101

Not on the map. Why?

Abstract (arXiv)

Communication compression, a technique aiming to reduce the information volume to be transmitted over the air, has gained great interests in Federated Learning (FL) for the potential of alleviating its communication overhead. However, communication compression brings forth new challenges in FL due to the interplay of compression-incurred information distortion and inherent characteristics of FL such as partial participation and data heterogeneity. Despite the recent development, the performance of compressed FL approaches has not been fully exploited. The existing approaches either cannot accommodate arbitrary data heterogeneity or partial participation, or require stringent conditions on compression. In this paper, we revisit the seminal stochastic controlled averaging method by proposing an equivalent but more efficient/simplified formulation with halved uplink communication costs. Building upon this implementation, we propose two compressed FL algorithms, SCALLION and SCAFCOM, to support unbiased and biased compression, respectively. Both the proposed methods outperform the existing compressed FL methods in terms of communication and computation complexities. Moreover, SCALLION and SCAFCOM accommodates arbitrary data heterogeneity and do not make any additional assumptions on compression errors. Experiments show that SCALLION and SCAFCOM can match the performance of corresponding full-precision FL approaches with substantially reduced uplink communication, and outperform recent compressed FL methods under the same communication budget.

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). Builds on 59 of its 72 references that are in the map. First posted on arXiv 16 Aug 2023.

Builds on

59 of 72 references.

References · 59

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  6. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  7. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  8. 2018Federated Learning with Non-IID Datacited by 1,108
  9. 2020Adaptive Federated Optimizationcited by 1,046
  10. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  11. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  12. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
38 more
  1. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  2. 2019Agnostic Federated Learningcited by 456
  3. 2018Sparsified SGD with Memorycited by 376
  4. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  5. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  6. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  7. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  8. 2018The Convergence of Sparsified Gradient Methodscited by 229
  9. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 221
  10. 2018Federated Learning via Over-the-Air Computationcited by 219
  11. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  12. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
  13. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  14. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 181
  15. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
  16. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 160
  17. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
  18. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
  19. 2020On Biased Compression for Distributed Learningcited by 140
  20. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 133
  21. 2020Optimal Client Sampling for Federated Learningcited by 124
  22. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  23. 2020An Improved Analysis of Stochastic Gradient Descent with Momentumcited by 119
  24. 2020PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimizationcited by 119
  25. 2019Natural Compression for Distributed Deep Learningcited by 90
  26. 2019Variance Reduced Local SGD with Lower Communication Complexitycited by 80
  27. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  28. 2021On Large-Cohort Training for Federated Learningcited by 74
  29. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  30. 2018A Unified Analysis of Stochastic Momentum Methods for Deep Learningcited by 55
  31. 2020Faster Non-Convex Federated Learning via Global and Local Momentumcited by 48
  32. 2020Uncertainty Principle for Communication Compression in Distributed and Federated Learning and the Search for an Optimal Compressorcited by 45
  33. 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
  34. 2022BEER: Fast O(1/T) Rate for Decentralized Nonconvex Optimization with Communication Compressioncited by 41
  35. 2021DecentLaM: Decentralized Momentum SGD for Large-batch Deep Trainingcited by 40
  36. 2022On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyondcited by 38
  37. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 38
  38. 2022Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compressioncited by 33

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 34; 2025: 35; 2026: 27.

101 papers.

Sort:

By area:

  1. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  2. 2024Asynchronous Local-SGD Training for Language Modelingcited by 26
  3. 2025FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averagingcited by 25
  4. 2024Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchorscited by 21
  5. 2023PFLlib: A Beginner-Friendly and Comprehensive Personalized Federated Learning Library and Benchmarkcited by 17
  6. 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
  7. 2024From Challenges and Pitfalls to Recommendations and Opportunities: Implementing Federated Learning in Healthcarecited by 16
  8. 2024Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimizationcited by 13
  9. 2025Overcoming the Communication-Performance Tradeoff in LLM Pretrainingcited by 12
  10. 2024LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compressioncited by 11
  11. 2026MM-OpenFGL: A Comprehensive Benchmark for Multimodal Federated Graph Learningcited by 11
  12. 2024Personalized Federated Learning for Spatio-Temporal Forecasting: A Dual Semantic Alignment-Based Contrastive Approachcited by 8
38 more
  1. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environmentscited by 8
  2. 2024Real World Federated Learning with a Knowledge Distilled Transformer for Cardiac CT Imagingcited by 8
  3. 2025Towards Trustworthy Federated Learning with Untrusted Participantscited by 8
  4. 2024Ferret: Federated Full-Parameter Tuning at Scale for Large Language Modelscited by 6
  5. 2024Why Go Full? Elevating Federated Learning Through Partial Network Updatescited by 6
  6. 2025Personalized Federated Fine-tuning for Heterogeneous Data: An Automatic Rank Learning Approach via Two-Level LoRAcited by 6
  7. 2025Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainablecited by 6
  8. 2024Towards Faster Decentralized Stochastic Optimization with Communication Compressioncited by 5
  9. 2025Interaction-Aware Gaussian Weighting for Clustered Federated Learningcited by 5
  10. 2023Leveraging Function Space Aggregation for Federated Learning at Scalecited by 4
  11. 2024Flashback: Understanding and Mitigating Forgetting in Federated Learningcited by 4
  12. 2024A New Theoretical Perspective on Data Heterogeneity in Federated Optimizationcited by 4
  13. 2024Towards Fast Rates for Federated and Multi-Task Reinforcement Learningcited by 4
  14. 2024NTK-DFL: Enhancing Decentralized Federated Learning in Heterogeneous Settings via Neural Tangent Kernelcited by 4
  15. 2025Revisiting LocalSGD and SCAFFOLD: Improved Rates and Missing Analysiscited by 4
  16. 2024Distributed Event-Based Learning via ADMMcited by 3
  17. 2024SPAM: Stochastic Proximal Point Method with Momentum Variance Reduction for Non-convex Cross-Device Federated Learningcited by 3
  18. 2024Towards Layer-Wise Personalized Federated Learning: Adaptive Layer Disentanglement via Conflicting Gradientscited by 3
  19. 2025Delayed Momentum Aggregation: Communication-efficient Byzantine-robust Federated Learning with Partial Participationcited by 3
  20. 2023Controlling Federated Learning for Covertnesscited by 2
  21. 2024An Auction-based Marketplace for Model Trading in Federated Learningcited by 2
  22. 2024FAGH: Accelerating Federated Learning with Approximated Global Hessiancited by 2
  23. 2024Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learningcited by 2
  24. 2025Communication-Efficient, 2D Parallel Stochastic Gradient Descent for Distributed-Memory Optimizationcited by 2
  25. 2025Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMscited by 2
  26. 2025Generalization in Federated Learning: A Conditional Mutual Information Frameworkcited by 2
  27. 2025FedHQ: Hybrid Runtime Quantization for Federated Learningcited by 2
  28. 2025The Actor-Critic Update Order Matters for PPO in Federated Reinforcement Learningcited by 2
  29. 2025Constant Stepsize Local GD for Logistic Regression: Acceleration by Instabilitycited by 2
  30. 2025FedOne: Query-Efficient Federated Learning for Black-box Discrete Prompt Learningcited by 2
  31. 2025FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Modelscited by 2
  32. 2025Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvgcited by 2
  33. 2025Convergence Guarantees for Federated SARSA with Local Training and Heterogeneous Agentscited by 2
  34. 2026FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuningcited by 2
  35. 2023Achieving Linear Speedup with ProxSkip in Distributed Stochastic Optimizationcited by 1
  36. 2024Buffer-based Gradient Projection for Continual Federated Learningcited by 1
  37. 2024DynamicFL: Federated Learning with Dynamic Communication Resource Allocationcited by 1
  38. 2024Fully First-Order Methods for Decentralized Bilevel Optimizationcited by 1

50 most cited of 101 All 101 (JSON)