mlsanityData to 14 Oct 2026

Coordinating Momenta for Cross-silo Federated Learning

arXiv 2102.03970 ↗First posted 8 Feb 2021cs.LG cs.DCcited by 9

Not on the map. Why?

Abstract (arXiv)

Communication efficiency is crucial for federated learning (FL). Conducting local training steps in clients to reduce the communication frequency between clients and the server is a common method to address this issue. However, this strategy leads to the client drift problem due to textitnon-i.i.d. data distributions in different clients which severely deteriorates the performance. In this work, we propose a new method to improve the training performance in cross-silo FL via maintaining double momentum buffers. In our algorithm, one momentum buffer is used to track the server model updating direction, and the other one is adopted to track the local model updating direction. More important, we introduce a novel momentum fusion technique to coordinate the server and local momentum buffers. We also derive the first theoretical convergence analysis involving both the server and local standard momentum SGD. Extensive deep FL experimental results verify that our new approach has a better training performance than the FedAvg and existing standard momentum SGD variants.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 34 of its 50 references that are in the map. First posted on arXiv 8 Feb 2021.

Builds on

34 of 50 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2019Advances and Open Problems in Federated Learningcited by 3,046
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  9. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  10. 2020Adaptive Federated Optimizationcited by 1,046
  11. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  12. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
22 more
  1. 2019Towards Federated Learning at Scale: System Designcited by 832
  2. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  3. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  4. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  5. 2019Agnostic Federated Learningcited by 456
  6. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
  7. 2019Fair Resource Allocation in Federated Learningcited by 394
  8. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  9. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  10. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  11. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 269
  12. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
  13. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 179
  14. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
  15. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  16. 2019Variance Reduced Local SGD with Lower Communication Complexitycited by 80
  17. 2018Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGDcited by 77
  18. 2018On the Ineffectiveness of Variance Reduced Optimization for Deep Learningcited by 48
  19. 2020Faster On-Device Training Using New Federated Momentum Algorithmcited by 25
  20. 2019On the Acceleration of Deep Learning Model Parallelism with Stalenesscited by 17
  21. 2020Step-Ahead Error Feedback for Distributed Training with Compressed Gradientcited by 9
  22. 2020Optimal Gradient Quantization Condition for Communication-Efficient Distributed Trainingcited by 1

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 2; 2024: 1; 2025: 3.

9 papers.

  1. 2022Faster Adaptive Federated Learningcited by 35
  2. 2023GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgettingcited by 27
  3. 2022SphereFed: Hyperspherical Federated Learningcited by 19
  4. 2022Hierarchical Federated Learning with Momentum Acceleration in Multi-Tier Networkscited by 6
  5. 2025Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practicecited by 3
  6. 2024FedECADO: A Dynamical System Model of Federated Learningcited by 2
  7. 2023Knowledge Rumination for Client Utility Evaluation in Heterogeneous Federated Learningcited by 0
  8. 2025Stratify: Rethinking Federated Learning for Non-IID Data through Balanced Samplingcited by 0
  9. 2025Adaptive Federated Learning via Dynamical System Modelcited by 0