mlsanityData to 14 Oct 2026

Generalization in Federated Learning: A Conditional Mutual Information Framework

arXiv 2503.04091 ↗First posted 6 Mar 2025stat.ML cs.IT cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is a widely adopted privacy-preserving distributed learning framework, yet its generalization performance remains less explored compared to centralized learning. In FL, the generalization error consists of two components: the out-of-sample gap, which measures the gap between the empirical and true risk for participating clients, and the participation gap, which quantifies the risk difference between participating and non-participating clients. In this work, we apply an information-theoretic analysis via the conditional mutual information (CMI) framework to study FL's two-level generalization. Beyond the traditional supersample-based CMI framework, we introduce a superclient construction to accommodate the two-level generalization setting in FL. We derive multiple CMI-based bounds, including hypothesis-based CMI bounds, illustrating how privacy constraints in FL can imply generalization guarantees. Furthermore, we propose fast-rate evaluated CMI bounds that recover the best-known convergence rate for two-level FL generalization in the small empirical risk regime. For specific FL model aggregation strategies and structured loss functions, we refine our bounds to achieve improved convergence rates with respect to the number of participating clients. Empirical evaluations confirm that our evaluated CMI bounds are non-vacuous and accurately capture the generalization behavior of FL algorithms.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 36 of its 59 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

36 of 59 references.

References · 36

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2020Adaptive Federated Optimizationcited by 1,046
  6. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  7. 2019Agnostic Federated Learningcited by 456
  8. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
  9. 2013The Composition Theorem for Differential Privacycited by 297
  10. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 147
  11. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
  12. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 119
24 more
  1. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  2. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  3. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 99
  4. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  5. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  6. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  7. 2021Information-theoretic generalization bounds for black-box learning algorithmscited by 49
  8. 2016Differential Privacy as a Mutual Information Constraintcited by 45
  9. 2022A Unified Analysis of Federated Learning with Arbitrary Client Participationcited by 45
  10. 2021Minibatch vs Local SGD with Shuffling: Tight Convergence Bounds and Beyondcited by 36
  11. 2022A New Family of Generalization Bounds Using Samplewise Evaluated CMIcited by 28
  12. 2023Understanding Generalization of Federated Learning via Stability: Heterogeneity Matterscited by 27
  13. 2023Tighter Information-Theoretic Generalization Bounds from Supersamplescited by 22
  14. 2020Information-Theoretic Bounds on the Generalization Error and Privacy Leakage in Federated Learningcited by 21
  15. 2022Information-Theoretic Analysis of Unsupervised Domain Adaptationcited by 15
  16. 2022Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learningcited by 14
  17. 2022Evaluated CMI Bounds for Meta Learning: Tightness and Expressivenesscited by 11
  18. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  19. 2023More Communication Does Not Result in Smaller Generalization Error in Federated Learningcited by 10
  20. 2023Lessons from Generalization Error Analysis of Federated Learning: You May Communicate Less Often!cited by 8
  21. 2023Sample-Conditioned Hypothesis Stability Sharpens Information-Theoretic Generalization Boundscited by 6
  22. 2023Comparing Comparators in Generalization Boundscited by 5
  23. 2024Differentiated Aggregation to Improve Generalization in Federated Learningcited by 4
  24. 2024Generalization Bounds via Conditional f-Informationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025An Information-Theoretic Analysis of OOD Generalization in Meta-Reinforcement Learningcited by 0
  2. 2026A Hierarchical Sampling Framework for bounding the Generalization Error of Federated Learningcited by 0