mlsanityData to 7 Oct 2026

More Communication Does Not Result in Smaller Generalization Error in Federated Learning

arXiv 2304.12216 ↗First posted 24 Apr 2023stat.ML cs.IT cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

We study the generalization error of statistical learning models in a Federated Learning (FL) setting. Specifically, there are K devices or clients, each holding an independent own dataset of size n. Individual models, learned locally via Stochastic Gradient Descent, are aggregated (averaged) by a central server into a global model and then sent back to the devices. We consider multiple (say R ∈ mathbb N^*) rounds of model aggregation and study the effect of R on the generalization error of the final aggregated model. We establish an upper bound on the generalization error that accounts explicitly for the effect of R (in addition to the number of participating devices K and dataset size n). It is observed that, for fixed (n, K), the bound increases with R, suggesting that the generalization of such learning algorithms is negatively affected by more frequent communication with the parameter server. Combined with the fact that the empirical risk, however, generally decreases for larger values of R, this indicates that R might be a parameter to optimize to reduce the population risk of FL algorithms. The results of this paper, which extend straightforwardly to the heterogeneous data setting, are also illustrated through numerical examples.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 18 of its 34 references that are in the map. First posted on arXiv 24 Apr 2023.

Builds on

18 of 34 references.

References · 18

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  5. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 528
  6. 2018Distributed learning of deep neural network over multiple agentscited by 277
  7. 2017Three Factors Influencing Minima in SGDcited by 270
  8. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  9. 2018A Hybrid Approach to Privacy-Preserving Federated Learningcited by 208
  10. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  11. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 106
  12. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
6 more
  1. 2020Information-Theoretic Bounds on the Generalization Error and Privacy Leakage in Federated Learningcited by 20
  2. 2022Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learningcited by 14
  3. 2022Generalization Error Bounds for Deep Neural Networks Trained by SGDcited by 14
  4. 2021In-Network Learning: Distributed Training and Inference in Networkscited by 8
  5. 2022Improved Information Theoretic Generalization Bounds for Distributed and Federated Learningcited by 8
  6. 2022Faster Convergence of Local SGD for Over-Parameterized Modelscited by 4

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 3; 2025: 3.

10 papers.

  1. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  2. 2023Lessons from Generalization Error Analysis of Federated Learning: You May Communicate Less Often!cited by 8
  3. 2023Federated PAC-Bayesian Learning on Non-IID datacited by 2
  4. 2023Federated Learning with Nonvacuous Generalisation Boundscited by 2
  5. 2025Generalization in Federated Learning: A Conditional Mutual Information Frameworkcited by 2
  6. 2024Bayesian Federated Model Compression for Communication and Computation Efficiencycited by 1
  7. 2024Accelerating Energy-Efficient Federated Learning in Cell-Free Networks with Adaptive Quantizationcited by 1
  8. 2025Heterogeneity Matters even More in Distributed Learning: Study from Generalization Perspectivecited by 1
  9. 2024Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?cited by 0
  10. 2025Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptationcited by 0