mlsanityData to 14 Oct 2026

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead

arXiv 2502.06349 ↗First posted 10 Feb 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated ensemble distillation addresses client heterogeneity by generating pseudo-labels for an unlabeled server dataset based on client predictions and training the server model using the pseudo-labeled dataset. The unlabeled server dataset can either be pre-existing or generated through a data-free approach. The effectiveness of this approach critically depends on the method of assigning weights to client predictions when creating pseudo-labels, especially in highly heterogeneous settings. Inspired by theoretical results from GANs, we propose a provably near-optimal weighting method that leverages client discriminators trained with a server-distributed generator and local datasets. Our experiments on various image classification tasks demonstrate that the proposed method significantly outperforms baselines. Furthermore, we show that the additional communication cost, client-side privacy leakage, and client-side computational overhead introduced by our method are negligible, both in scenarios with and without a pre-existing server dataset.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 43 of its 73 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

43 of 73 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2019Advances and Open Problems in Federated Learningcited by 3,046
  9. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  10. 2017Improved Training of Wasserstein GANscited by 2,235
  11. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  12. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
31 more
  1. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  2. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  3. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  4. 2020Federated Learning with Matched Averagingcited by 564
  5. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 492
  6. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  7. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  8. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  9. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 93
  10. 2020FedGAN: Federated Generative Adversarial Networks for Distributed Datacited by 90
  11. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  12. 2019Variance Reduced Local SGD with Lower Communication Complexitycited by 80
  13. 2022Tackling Data Heterogeneity in Federated Learning with Class Prototypescited by 67
  14. 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 67
  15. 2021Personalized Federated Learning through Local Memorizationcited by 57
  16. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
  17. 2022Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learningcited by 52
  18. 2018MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasetscited by 49
  19. 2024FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learningcited by 47
  20. 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
  21. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Datacited by 31
  22. 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
  23. 2020Communication-Efficient Federated Distillationcited by 28
  24. 2024Do Generated Data Always Help Contrastive Learning?cited by 25
  25. 2022FedMed-GAN: Federated Domain Translation on Unsupervised Cross-Modality Brain Image Synthesiscited by 21
  26. 2022Improving GANs with A Dynamic Discriminatorcited by 20
  27. 2020Federated Generative Adversarial Learningcited by 19
  28. 2024FedImpro: Measuring and Improving Client Update in Federated Learningcited by 15
  29. 2021An Efficient Federated Distillation Learning System for Multi-task Time Series Classificationcited by 14
  30. 2023FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learningcited by 8
  31. 2019Investigating Under and Overfitting in Wasserstein Generative Adversarial Networkscited by 7

Built on by

Not cited yet.