mlsanityData to 7 Oct 2026

FedGuCci: Making Local Models More Connected in Landscape for Federated Learning

arXiv 2402.18949 ↗First posted 29 Feb 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) involves multiple heterogeneous clients collaboratively training a global model via iterative local updates and model fusion. The generalization of FL's global model has a large gap compared with centralized training, which is its bottleneck for broader applications. In this paper, we study and improve FL's generalization through a fundamental ``connectivity'' perspective, which means how the local models are connected in the parameter region and fused into a generalized global model. The term ``connectivity'' is derived from linear mode connectivity (LMC), studying the interpolated loss landscape of two different solutions (e.g., modes) of neural networks. Bridging the gap between LMC and FL, in this paper, we leverage fixed anchor models to empirically and theoretically study the transitivity property of connectivity from two models (LMC) to a group of models (model fusion in FL). Based on the findings, we propose FedGuCci(+), improving group connectivity for better generalization. It is shown that our methods can boost the generalization of FL under client heterogeneity across various tasks (4 CV datasets and 6 NLP datasets) and model architectures (e.g., ViTs and PLMs). The code is available here: hrefhttps://github.com/ZexiLee/fedguccifaGithub~FedGuCci Codebase.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 64 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

43 of 64 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  12. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
31 more
  1. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  2. 2018Neural Network Acceptability Judgmentscited by 952
  3. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  4. 2021Model-Contrastive Federated Learningcited by 762
  5. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  6. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  7. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  8. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  9. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  10. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  11. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  12. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  13. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  14. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  15. 2020Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustnesscited by 130
  16. 2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 108
  17. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  18. 2021Learning Neural Network Subspacescited by 82
  19. 2021On Large-Cohort Training for Federated Learningcited by 73
  20. 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 66
  21. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  22. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  23. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  24. 2021What can linear interpolation of neural network loss landscapes tell us?cited by 27
  25. 2022Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matchingcited by 24
  26. 2021Connecting Low-Loss Subspace for Personalized Federated Learningcited by 12
  27. 2023Learning Cautiously in Federated Learning with Noisy and Heterogeneous Clientscited by 12
  28. 2024Improving Model Fusion by Training-time Neuron Alignment with Fixed Neuron Anchorscited by 7
  29. 2023Edge-cloud Collaborative Learning with Federated and Centralized Featurescited by 5
  30. 2023FediOS: Decoupling Orthogonal Subspaces for Personalization in Feature-skew Federated Learningcited by 5
  31. 2023Mode Connectivity and Data Heterogeneity of Federated Learningcited by 3

Built on by

Not cited yet.