mlsanityData to 7 Oct 2026

FedCos: A Scene-adaptive Federated Optimization Enhancement for Performance Improvement

arXiv 2204.03174 ↗First posted 7 Apr 2022cs.LG cs.AI cs.DCcited by 1

Not on the map. Why?

Abstract (arXiv)

As an emerging technology, federated learning (FL) involves training machine learning models over distributed edge devices, which attracts sustained attention and has been extensively studied. However, the heterogeneity of client data severely degrades the performance of FL compared with that in centralized training. It causes the locally trained models of clients to move in different directions. On the one hand, it slows down or even stalls the global updates, leading to inefficient communication. On the other hand, it enlarges the distances between local models, resulting in an aggregated global model with poor performance. Fortunately, these shortcomings can be mitigated by reducing the angle between the directions that local models move in. Based on this fact, we propose FedCos, which reduces the directional inconsistency of local models by introducing a cosine-similarity penalty. It promotes the local model iterations towards an auxiliary global direction. Moreover, our approach is auto-adapt to various non-IID settings without an elaborate selection of hyperparameters. The experimental results show that FedCos outperforms the well-known baselines and can enhance them under a variety of FL scenes, including varying degrees of data heterogeneity, different number of participants, and cross-silo and cross-device settings. Besides, FedCos improves communication efficiency by 2 to 5 times. With the help of FedCos, multiple FL methods require significantly fewer communication rounds than before to obtain a model with comparable performance.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 7 Apr 2022.

Builds on

31 of 45 references.

References · 31

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  6. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  7. 2018Federated Learning with Non-IID Datacited by 1,099
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  10. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  11. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  12. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
19 more
  1. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  2. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  3. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  4. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
  5. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  6. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
  7. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  8. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  9. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  10. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  11. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 150
  12. 2021Distributionally Robust Federated Averagingcited by 104
  13. 2020Robust Federated Learning: The Case of Affine Distribution Shiftscited by 102
  14. 2020Defending against Backdoors in Federated Learning with Robust Learning Ratecited by 82
  15. 2020Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithmscited by 78
  16. 2020Faster On-Device Training Using New Federated Momentum Algorithmcited by 25
  17. 2019Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costscited by 12
  18. 2019Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descentcited by 7
  19. 2021Federated Learning-based Active Authentication on Mobile Devicescited by 7

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022CC-FedAvg: Computationally Customized Federated Averagingcited by 2