Mode Connectivity and Data Heterogeneity of Federated Learning
Not on the map. Why?
Abstract (arXiv)
Federated learning (FL) enables multiple clients to train a model while keeping their data private collaboratively. Previous studies have shown that data heterogeneity between clients leads to drifts across client updates. However, there are few studies on the relationship between client and global modes, making it unclear where these updates end up drifting. We perform empirical and theoretical studies on this relationship by utilizing mode connectivity, which measures performance change (i.e., connectivity) along parametric paths between different modes. Empirically, reducing data heterogeneity makes the connectivity on different paths more similar, forming more low-error overlaps between client and global modes. We also find that a barrier to connectivity occurs when linearly connecting two global modes, while it disappears with considering non-linear mode connectivity. Theoretically, we establish a quantitative bound on the global-mode connectivity using mean-field theory or dropout stability. The bound demonstrates that the connectivity improves when reducing data heterogeneity and widening trained models. Numerical results further corroborate our analytical findings.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 21 of its 35 references that are in the map. First posted on arXiv 29 Sep 2023.
Builds on
21 of 35 references.
References · 21
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2019Advances and Open Problems in Federated Learningcited by 3,046
- 2018Federated Optimization in Heterogeneous Networkscited by 2,987
- 2018Federated Learning with Non-IID Datacited by 1,108
- 2017Deeper, Broader and Artier Domain Generalizationcited by 986
- 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
- 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
- 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
- 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
- 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
9 more
- 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
- 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
- 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
- 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
- 2019Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Netscited by 63
- 2022FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Datacited by 50
- 2019Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networkscited by 35
- 2023A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiencycited by 21
- 2023Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Datacited by 7
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 1.
3 papers.