# Revisiting Ensembling in One-Shot Federated Learning

arXiv: https://arxiv.org/abs/2411.07182 · HTML: https://mlsanity.com/paper/2411.07182/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 51 references that are in the map.
- First posted on arXiv 11 Nov 2024.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Differentially Private Federated Learning: A Client Level Perspective](https://mlsanity.com/paper/1712.07557/) (2017)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021)
- [FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning](https://mlsanity.com/paper/2009.01974/) (2020)
- [One-Shot Federated Learning](https://mlsanity.com/paper/1902.11175/) (2019)
- [FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings](https://mlsanity.com/paper/2210.04620/) (2022)
- [Distilled One-Shot Federated Learning](https://mlsanity.com/paper/2009.07999/) (2020)
- [DENSE: Data-Free One-Shot Federated Learning](https://mlsanity.com/paper/2112.12371/) (2021)
- [Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning](https://mlsanity.com/paper/2204.12703/) (2022)
- [Practical One-Shot Federated Learning for Cross-Silo Setting](https://mlsanity.com/paper/2010.01017/) (2020)
- [Differentially Private Federated Learning on Heterogeneous Data](https://mlsanity.com/paper/2111.09278/) (2021)

## Built on by (most cited first, 13 of 13)

- [One-shot Federated Learning Methods: A Practical Guide](https://mlsanity.com/paper/2502.09104/) (2025)
- [FedBEns: One-Shot Federated Learning based on Bayesian Ensemble](https://mlsanity.com/paper/2503.15367/) (2025)
- [The Gaussian-Head OFL Family: One-Shot Federated Learning from Client Global Statistics](https://mlsanity.com/paper/2602.01186/) (2026)
- [Towards One-shot Federated Learning: Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2505.02426/) (2025)
- [Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learning](https://mlsanity.com/paper/2509.22700/) (2025)
- [Robust Federated Inference](https://mlsanity.com/paper/2510.00310/) (2025)
- [Efficient Federated Search for Retrieval-Augmented Generation using Lightweight Routing](https://mlsanity.com/paper/2502.19280/) (2025)
- [Distribution Alignment for One-Shot Federated Learning via Optimal Transport](https://mlsanity.com/paper/2606.16655/) (2026)
- [Single-Round Scalable Analytic Federated Learning](https://mlsanity.com/paper/2512.03336/) (2025)
- [Learned Neighbor Trust for Collaborative Deployment in Model-Agnostic Decentralized Learning](https://mlsanity.com/paper/2605.05009/) (2026)
- [Auto-FL-Research: Agentic Search for Federated Learning Algorithms](https://mlsanity.com/paper/2607.01366/) (2026)
- [CRIP: Channel Level Representation Injection for Personalized One-Shot Federated Learning](https://mlsanity.com/paper/2608.02222/) (2026)
- [Structural Entropy-Driven Graph Diffusion Generation for One-Shot Federated Graph Learning](https://mlsanity.com/paper/2609.06499/) (2026)

## Abstract (arXiv)

Federated learning (FL) is an appealing approach to training machine learning models without sharing raw data. However, standard FL algorithms are iterative and thus induce a significant communication cost. One-shot federated learning (OFL) trades the iterative exchange of models between clients and the server with a single round of communication, thereby saving substantially on communication costs. Not surprisingly, OFL exhibits a performance gap in terms of accuracy with respect to FL, especially under high data heterogeneity. We introduce FENS, a novel federated ensembling scheme that approaches the accuracy of FL with the communication efficiency of OFL. Learning in FENS proceeds in two phases: first, clients train models locally and send them to the server, similar to OFL; second, clients collaboratively train a lightweight prediction aggregator model using FL. We showcase the effectiveness of FENS through exhaustive experiments spanning several datasets and heterogeneity levels. In the particular case of heterogeneously distributed CIFAR-10 dataset, FENS achieves up to a 26.9% higher accuracy over state-of-the-art (SOTA) OFL, being only 3.1% lower than FL. At the same time, FENS incurs at most 4.3x more communication than OFL, whereas FL is at least 10.9x more communication-intensive than FENS.