mlsanityData to 14 Oct 2026

What Do We Mean by Generalization in Federated Learning?

arXiv 2110.14216 ↗First posted 27 Oct 2021cs.LG cs.DC stat.MLcited by 62

Not on the map. Why?

Abstract (arXiv)

Federated learning data is drawn from a distribution of distributions: clients are drawn from a meta-distribution, and their data are drawn from local data distributions. Thus generalization studies in federated learning should separate performance gaps from unseen client data (out-of-sample gap) from performance gaps from unseen client distributions (participation gap). In this work, we propose a framework for disentangling these performance gaps. Using this framework, we observe and explain differences in behavior across natural and synthetic federated datasets, indicating that dataset synthesis strategy can be important for realistic simulations of generalization in federated learning. We propose a semantic synthesis strategy that enables realistic simulation without naturally-partitioned data. Informed by our findings, we call out community suggestions for future federated learning works.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 77 of its 105 references that are in the map. First posted on arXiv 27 Oct 2021.

Builds on

77 of 105 references.

References · 77

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2016Concrete Problems in AI Safetycited by 1,642
  7. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  8. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  9. 2018Group Normalizationcited by 1,524
  10. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  11. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  12. 2018Federated Learning with Non-IID Datacited by 1,108
38 more
  1. 2020Adaptive Federated Optimizationcited by 1,046
  2. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  3. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  4. 2018LEAF: A Benchmark for Federated Settingscited by 792
  5. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  6. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  7. 2017Federated Multi-Task Learningcited by 672
  8. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  9. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  10. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  11. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  12. 2015Importance Weighted Autoencoderscited by 456
  13. 2019Agnostic Federated Learningcited by 456
  14. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 417
  15. 2019Fair Resource Allocation in Federated Learningcited by 394
  16. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 367
  17. 2020Adaptive Personalized Federated Learningcited by 358
  18. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  19. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 325
  20. 2021A Field Guide to Federated Optimizationcited by 315
  21. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  22. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 269
  23. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  24. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  25. 2017EMNIST: an extension of MNIST to handwritten letterscited by 213
  26. 2020Federated Learning of a Mixture of Global and Local Modelscited by 210
  27. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  28. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 181
  29. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
  30. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  31. 2018Federated Meta-Learning with Fast Convergence and Efficient Communicationcited by 155
  32. 2019Federated Evaluation of On-device Personalizationcited by 147
  33. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
  34. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
  35. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  36. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 142
  37. 2020On Biased Compression for Distributed Learningcited by 140
  38. 2020A Principled Approach to Data Valuation for Federated Learningcited by 127

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 13; 2023: 13; 2024: 19; 2025: 13; 2026: 3.

62 papers.

Sort:

By area:

  1. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  2. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  3. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 99
  4. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  5. 2022Federated Learning on Non-IID Graphs via Structural Knowledge Sharingcited by 88
  6. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  7. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  8. 2022pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learningcited by 44
  9. 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
  10. 2023Efficient Personalized Federated Learning via Sparse Model-Adaptationcited by 38
  11. 2022Motley: Benchmarking Heterogeneity and Personalization in Federated Learningcited by 31
  12. 2022Accelerated Federated Learning with Decoupled Adaptive Optimizationcited by 23
38 more
  1. 2022Gradient Masked Averaging for Federated Learningcited by 22
  2. 2023FedFA: Federated Feature Augmentationcited by 22
  3. 2024FedImpro: Measuring and Improving Client Update in Federated Learningcited by 15
  4. 2022Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learningcited by 14
  5. 2023Benchmarking Algorithms for Federated Domain Generalizationcited by 14
  6. 2024FOOGD: Federated Collaboration for Both Out-of-distribution Generalization and Detectioncited by 14
  7. 2024Federated Learning with Bilateral Curation for Partially Class-Disjoint Datacited by 13
  8. 2024FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusioncited by 11
  9. 2022Auxo: Efficient Federated Learning via Scalable Client Clusteringcited by 8
  10. 2023Making Batch Normalization Great in Federated Deep Learningcited by 8
  11. 2022On Noisy Evaluation in Federated Hyperparameter Tuningcited by 7
  12. 2023Federated Learning under Covariate Shifts with Generalization Guaranteescited by 6
  13. 2024PraFFL: A Preference-Aware Scheme in Fair Federated Learningcited by 5
  14. 2024Federated brain tumor segmentation: an extensive benchmarkcited by 5
  15. 2025FAA-CLIP: Federated Adversarial Adaptation of CLIPcited by 5
  16. 2022Wasserstein Distributionally Robust Optimization with Wasserstein Barycenterscited by 3
  17. 2023Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learningcited by 3
  18. 2024FedNS: A Fast Sketching Newton-Type Algorithm for Federated Learningcited by 3
  19. 2024Boosting Federated Domain Generalization: Understanding the Role of Advanced Pre-Trained Architecturescited by 3
  20. 2024Communication-Efficient Federated Group Distributionally Robust Optimizationcited by 3
  21. 2025pMixFed: Efficient Personalized Federated Learning through Adaptive Layer-Wise Mixupcited by 3
  22. 2022Federated Training of Dual Encoding Models on Small Non-IID Client Datasetscited by 2
  23. 2023Federated Variational Inference: Towards Improved Personalization and Generalizationcited by 2
  24. 2023Towards Federated Learning Under Resource Constraints via Layer-wise Training and Depth Dropoutcited by 2
  25. 2024FedRSU: Federated Learning for Scene Flow Estimation on Roadside Unitscited by 2
  26. 2024How to Collaborate: Towards Maximizing the Generalization Performance in Cross-Silo Federated Learningcited by 2
  27. 2025FedDAG: Federated Domain Adversarial Generation Towards Generalizable Medical Image Analysiscited by 2
  28. 2025Generalization in Federated Learning: A Conditional Mutual Information Frameworkcited by 2
  29. 2025Federated Adapter on Foundation Models: An Out-Of-Distribution Approachcited by 2
  30. 2025FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Modelscited by 2
  31. 2023Advocating for the Silent: Enhancing Federated Generalization for Non-Participating Clientscited by 1
  32. 2023Whole-brain radiomics for clustered federated personalization in brain tumor segmentationcited by 1
  33. 2024Reducing Spurious Correlation for Federated Domain Generalizationcited by 1
  34. 2024Federated Domain Generalization via Prompt Learning and Aggregationcited by 1
  35. 2025Heterogeneity Matters even More in Distributed Learning: Study from Generalization Perspectivecited by 1
  36. 2025Federated Learning via Meta-Variational Dropoutcited by 1
  37. 2026GEN-Guard: Correcting Generalization Failures for Deployable Federated Surgical AIcited by 1
  38. 2024FedGuCci: Making Local Models More Connected in Landscape for Federated Learningcited by 0

50 most cited of 62 All 62 (JSON)