Heterogeneity Matters even More in Distributed Learning: Study from Generalization Perspective
Not on the map. Why?
Abstract (arXiv)
In this paper, we investigate the effect of data heterogeneity across clients on the performance of distributed learning systems, i.e., one-round Federated Learning, as measured by the associated generalization error. Specifically, K clients have each n training samples generated independently according to a possibly different data distribution, and their individually chosen models are aggregated by a central server. We study the effect of the discrepancy between the clients' data distributions on the generalization error of the aggregated model. First, we establish in-expectation and tail upper bounds on the generalization error in terms of the distributions. In part, the bounds extend the popular Conditional Mutual Information (CMI) bound, which was developed for the centralized learning setting, i.e., K=1, to the distributed learning setting with an arbitrary number of clients K ≥ 1. Then, we connect with information-theoretic rate-distortion theory to derive possibly tighter textitlossy versions of these bounds. Next, we apply our lossy bounds to study the effect of data heterogeneity across clients on the generalization error for the distributed classification problem in which each client uses Support Vector Machines (DSVM). In this case, we establish explicit generalization error bounds that depend explicitly on the data heterogeneity degree. It is shown that the bound gets smaller as the degree of data heterogeneity across clients increases, thereby suggesting that DSVM generalizes better when the dissimilarity between the clients' training samples is bigger. This finding, which goes beyond DSVM, is validated experimentally through several experiments.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 3 Mar 2025.
Builds on
35 of 47 references.
References · 35
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2019Advances and Open Problems in Federated Learningcited by 3,046
- 2018Federated Optimization in Heterogeneous Networkscited by 2,987
- 2016Deep Learning with Differential Privacycited by 2,728
- 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
- 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
- 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
- 2017Federated Multi-Task Learningcited by 672
- 2019Federated Learning with Personalization Layerscited by 541
- 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
- 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
- 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 459
23 more
- 2019Agnostic Federated Learningcited by 456
- 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
- 2018Distributed learning of deep neural network over multiple agentscited by 279
- 2017Sparse Communication for Distributed Gradient Descentcited by 249
- 2020Is Local SGD Better than Minibatch SGD?cited by 151
- 2020Reasoning About Generalization via Conditional Mutual Informationcited by 147
- 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
- 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
- 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
- 2021What Do We Mean by Generalization in Federated Learning?cited by 62
- 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Datacited by 38
- 2018Distributed Variational Representation Learningcited by 32
- 2023Understanding Generalization of Federated Learning via Stability: Heterogeneity Matterscited by 27
- 2022Rate-Distortion Theoretic Generalization Bounds for Stochastic Learning Algorithmscited by 22
- 2020Information-Theoretic Bounds on the Generalization Error and Privacy Leakage in Federated Learningcited by 21
- 2020Near-Tight Margin-Based Generalization Bounds for Support Vector Machinescited by 19
- 2021Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?cited by 15
- 2022Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learningcited by 14
- 2024Minimum Description Length and Generalization Guarantees for Representation Learningcited by 11
- 2023More Communication Does Not Result in Smaller Generalization Error in Federated Learningcited by 10
- 2023Data-dependent Generalization Bounds via Variable-Size Compressibilitycited by 9
- 2021In-Network Learning: Distributed Training and Inference in Networkscited by 8
- 2023Lessons from Generalization Error Analysis of Federated Learning: You May Communicate Less Often!cited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.