mlsanityData to 7 Oct 2026

Improved Information Theoretic Generalization Bounds for Distributed and Federated Learning

arXiv 2202.02423 ↗First posted 4 Feb 2022cs.IT cs.LG math.ITcited by 8

Not on the map. Why?

Abstract (arXiv)

We consider information-theoretic bounds on expected generalization error for statistical learning problems in a networked setting. In this setting, there are K nodes, each with its own independent dataset, and the models from each node have to be aggregated into a final centralized model. We consider both simple averaging of the models as well as more complicated multi-round algorithms. We give upper bounds on the expected generalization error for a variety of problems, such as those with Bregman divergence or Lipschitz continuous losses, that demonstrate an improved dependence of 1/K on the number of nodes. These "per node" bounds are in terms of the mutual information between the training dataset and the trained weights at each node, and are therefore useful in describing the generalization properties inherent to having communication or privacy constraints at each node.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 15 of its 26 references that are in the map. First posted on arXiv 4 Feb 2022.

Builds on

15 of 26 references.

References · 15

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  3. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  4. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  5. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  6. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  7. 2019High probability generalization bounds for uniformly stable algorithms with nearly optimal ratecited by 97
  8. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  9. 2018Generalization Error Bounds for Noisy, Iterative Algorithmscited by 79
  10. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  11. 2019Generalization Error Bounds Via R'enyi-, f-Divergences and Maximal Leakagecited by 48
  12. 2016Differential Privacy as a Mutual Information Constraintcited by 41
3 more
  1. 2020rTop-k: A Statistical Estimation Approach to Distributed SGDcited by 27
  2. 2020Information-Theoretic Bounds on the Generalization Error and Privacy Leakage in Federated Learningcited by 20
  3. 2021Generalization Bounds for Noisy Iterative Algorithms Using Properties of Additive Noise Channelscited by 15

Built on by

Cited by year

Citing papers per year: 2023: 6; 2024: 1; 2025: 1.

8 papers.

  1. 2023GPT-FL: Generative Pre-trained Model-Assisted Federated Learningcited by 24
  2. 2023More Communication Does Not Result in Smaller Generalization Error in Federated Learningcited by 10
  3. 2023Lessons from Generalization Error Analysis of Federated Learning: You May Communicate Less Often!cited by 8
  4. 2023Exactly Tight Information-Theoretic Generalization Error Bound for the Quadratic Gaussian Problemcited by 4
  5. 2024Differentiated Aggregation to Improve Generalization in Federated Learningcited by 4
  6. 2023Federated Learning with Nonvacuous Generalisation Boundscited by 2
  7. 2023Information-Theoretic Generalization Analysis for Topology-aware Heterogeneous Federated Edge Learning over Noisy Channelscited by 2
  8. 2025An Information-Theoretic Analysis for Federated Learning under Concept Driftcited by 2