mlsanityData to 7 Oct 2026

Revisiting Ensembling in One-Shot Federated Learning

arXiv 2411.07182 ↗First posted 11 Nov 2024cs.LG cs.DCcited by 13

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is an appealing approach to training machine learning models without sharing raw data. However, standard FL algorithms are iterative and thus induce a significant communication cost. One-shot federated learning (OFL) trades the iterative exchange of models between clients and the server with a single round of communication, thereby saving substantially on communication costs. Not surprisingly, OFL exhibits a performance gap in terms of accuracy with respect to FL, especially under high data heterogeneity. We introduce FENS, a novel federated ensembling scheme that approaches the accuracy of FL with the communication efficiency of OFL. Learning in FENS proceeds in two phases: first, clients train models locally and send them to the server, similar to OFL; second, clients collaboratively train a lightweight prediction aggregator model using FL. We showcase the effectiveness of FENS through exhaustive experiments spanning several datasets and heterogeneity levels. In the particular case of heterogeneously distributed CIFAR-10 dataset, FENS achieves up to a 26.9% higher accuracy over state-of-the-art (SOTA) OFL, being only 3.1% lower than FL. At the same time, FENS incurs at most 4.3x more communication than OFL, whereas FL is at least 10.9x more communication-intensive than FENS.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 11 Nov 2024.

Builds on

36 of 51 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2019Advances and Open Problems in Federated Learningcited by 3,021
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  9. 2016Deep Learning with Differential Privacycited by 2,682
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  12. 2020Adaptive Federated Optimizationcited by 1,036
24 more
  1. 2019Machine Unlearningcited by 1,020
  2. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  3. 2019Towards Federated Learning at Scale: System Designcited by 826
  4. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  5. 2020Federated Learning with Matched Averagingcited by 564
  6. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  7. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
  8. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  9. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  10. 2021Does Knowledge Distillation Really Work?cited by 187
  11. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  12. 2019One-Shot Federated Learningcited by 124
  13. 2022FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settingscited by 121
  14. 2020Distilled One-Shot Federated Learningcited by 108
  15. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  16. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  17. 2020Practical One-Shot Federated Learning for Cross-Silo Settingcited by 76
  18. 2021Differentially Private Federated Learning on Heterogeneous Datacited by 74
  19. 2020XOR Mixup: Privacy-Preserving Data Augmentation for One-Shot Federated Learningcited by 58
  20. 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillationcited by 45
  21. 2018Learn to Combine Modalities in Multimodal Deep Learningcited by 29
  22. 2024Enhancing One-Shot Federated Learning Through Data and Ensemble Co-Boostingcited by 27
  23. 2022Weighted Distillation with Unlabeled Examplescited by 12
  24. 2022Shielding Federated Learning Systems against Inference Attacks with ARM TrustZonecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 6.

13 papers.

Sort:

By area:

  1. 2025One-shot Federated Learning Methods: A Practical Guidecited by 6
  2. 2025FedBEns: One-Shot Federated Learning based on Bayesian Ensemblecited by 3
  3. 2025Towards One-shot Federated Learning: Advances, Challenges, and Future Directionscited by 2
  4. 2025Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learningcited by 2
  5. 2025Robust Federated Inferencecited by 2
  6. 2026The Gaussian-Head OFL Family: One-Shot Federated Learning from Client Global Statisticscited by 2
  7. 2025Efficient Federated Search for Retrieval-Augmented Generation using Lightweight Routingcited by 1
  8. 2026Distribution Alignment for One-Shot Federated Learning via Optimal Transportcited by 1
  9. 2025Single-Round Scalable Analytic Federated Learningcited by 0
  10. 2026Learned Neighbor Trust for Collaborative Deployment in Model-Agnostic Decentralized Learningcited by 0
  11. 2026Auto-FL-Research: Agentic Search for Federated Learning Algorithmscited by 0
  12. 2026CRIP: Channel Level Representation Injection for Personalized One-Shot Federated Learningcited by 0
1 more
  1. 2026Structural Entropy-Driven Graph Diffusion Generation for One-Shot Federated Graph Learningcited by 0