mlsanityData to 7 Oct 2026

Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning

arXiv 2107.10663 ↗First posted 21 Jul 2021stat.ML cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

In this paper we propose Fed-ensemble: a simple approach that bringsmodel ensembling to federated learning (FL). Instead of aggregating localmodels to update a single global model, Fed-ensemble uses random permutations to update a group of K models and then obtains predictions through model averaging. Fed-ensemble can be readily utilized within established FL methods and does not impose a computational overhead as it only requires one of the K models to be sent to a client in each communication round. Theoretically, we show that predictions on newdata from all K models belong to the same predictive posterior distribution under a neural tangent kernel regime. This result in turn sheds light onthe generalization advantages of model averaging. We also illustrate thatFed-ensemble has an elegant Bayesian interpretation. Empirical results show that our model has superior performance over several FL algorithms,on a wide range of data sets, and excels in heterogeneous settings often encountered in FL applications.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 44 of its 65 references that are in the map. First posted on arXiv 21 Jul 2021.

Builds on

44 of 65 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2019Advances and Open Problems in Federated Learningcited by 3,021
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  9. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  10. 2020Adaptive Federated Optimizationcited by 1,036
  11. 2019Towards Federated Learning at Scale: System Designcited by 826
  12. 2018LEAF: A Benchmark for Federated Settingscited by 780
32 more
  1. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  2. 2017Federated Multi-Task Learningcited by 669
  3. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  4. 2020The Future of Digital Health with Federated Learningcited by 625
  5. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  6. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  8. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  9. 2020Federated Learning with Matched Averagingcited by 564
  10. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  11. 2019Federated Learning with Personalization Layerscited by 533
  12. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  13. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  14. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  15. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  16. 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 347
  17. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
  18. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  19. 2020Federated Learning of a Mixture of Global and Local Modelscited by 208
  20. 2018Bayesian Model-Agnostic Meta-Learningcited by 194
  21. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  22. 2019Federated Evaluation of On-device Personalizationcited by 147
  23. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
  24. 2019Distributed Learning with Compressed Gradient Differencescited by 110
  25. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  26. 2020Federated Accelerated Stochastic Gradient Descentcited by 85
  27. 2018On Markov Chain Gradient Descentcited by 63
  28. 2020Uncertainty in Gradient Boosting via Ensemblescited by 56
  29. 2019A Double Residual Compression Algorithm for Efficient Distributed Learningcited by 31
  30. 2018Walkman: A Communication-Efficient Random-Walk Algorithm for Decentralized Optimizationcited by 30
  31. 2019The asymptotic spectrum of the Hessian of DNN throughout trainingcited by 19
  32. 2021FL-NTK: A Neural Tangent Kernel-based Framework for Federated Learning Convergence Analysiscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 6; 2024: 4; 2025: 4; 2026: 1.

23 papers.

Sort:

  1. 2022Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customizationcited by 34
  2. 2022Disentangled Federated Learning for Tackling Attributes Skew via Invariant Aggregation and Diversity Transferringcited by 28
  3. 2023FedACK: Federated Adversarial Contrastive Knowledge Distillation for Cross-Lingual and Cross-Model Social Bot Detectioncited by 12
  4. 2022Federated Data Analytics: A Study on Linear Modelscited by 9
  5. 2024Privacy Preserving Federated Learning in Medical Imaging with Uncertainty Estimationcited by 9
  6. 2021Federated Gaussian Process: Convergence, Automatic Personalization and Multi-fidelity Modelingcited by 8
  7. 2024Collaborative and Federated Black-box Optimization: A Bayesian Optimization Perspectivecited by 5
  8. 2022FEL: High Capacity Learning for Recommendation and Ranking via Federated Ensemble Learningcited by 4
  9. 2023Quality-Adaptive Split-Federated Learning for Segmenting Medical Images with Inaccurate Annotationscited by 4
  10. 2022Swan: A Neural Engine for Efficient DNN Training on Smartphone SoCscited by 3
  11. 2023Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectivescited by 3
  12. 2024Federated Learning over Connected Modescited by 3
11 more
  1. 2022FedCL: Federated Multi-Phase Curriculum Learning to Synchronously Correlate User Heterogeneitycited by 2
  2. 2023Benchmarking Collaborative Learning Methods Cost-Effectiveness for Prostate Segmentationcited by 2
  3. 2024Future-Proofing Medical Imaging with Privacy-Preserving Federated Learning and Uncertainty Quantification: A Reviewcited by 2
  4. 2025Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvgcited by 2
  5. 2023FedSDD: Scalable and Diversity-enhanced Distillation for Model Aggregation in Federated Learningcited by 1
  6. 2022GTFLAT: Game Theory Based Add-On For Empowering Federated Learning Aggregation Techniquescited by 0
  7. 2023Learning Driver Models for Automated Vehicles via Knowledge Sharing and Personalizationcited by 0
  8. 2025Query-based Knowledge Transfer for Heterogeneous Learning Environmentscited by 0
  9. 2025Stratify: Rethinking Federated Learning for Non-IID Data through Balanced Samplingcited by 0
  10. 2025SHEFL: Sparse Heterogeneous Ensemble Federated Learning with Group-Balanced Aggregationcited by 0
  11. 2026Federated Ensemble Learning with Progressive Model Personalizationcited by 0