mlsanityData to 7 Oct 2026

Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation

arXiv 2209.04599 ↗First posted 10 Sep 2022cs.CR cs.CV cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) is a machine learning paradigm where local nodes collaboratively train a central model while the training data remains decentralized. Existing FL methods typically share model parameters or employ co-distillation to address the issue of unbalanced data distribution. However, they suffer from communication bottlenecks. More importantly, they risk privacy leakage. In this work, we develop a privacy preserving and communication efficient method in a FL framework with one-shot offline knowledge distillation using unlabeled, cross-domain public data. We propose a quantized and noisy ensemble of local predictions from completely trained local models for stronger privacy guarantees without sacrificing accuracy. Based on extensive experiments on image classification and text classification tasks, we show that our privacy-preserving method outperforms baseline FL algorithms with superior performance in both accuracy and communication efficiency.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 43 of its 60 references that are in the map. First posted on arXiv 10 Sep 2022.

Builds on

43 of 60 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  9. 2016Pyramid Scene Parsing Networkcited by 2,472
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  12. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
31 more
  1. 2018Federated Learning with Non-IID Datacited by 1,099
  2. 2019Deep Leakage from Gradientscited by 1,004
  3. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  4. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  5. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
  6. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  7. 2017Federated Multi-Task Learningcited by 669
  8. 2020Federated Learning with Matched Averagingcited by 564
  9. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 508
  10. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  11. 2019Fair Resource Allocation in Federated Learningcited by 389
  12. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  13. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  14. 20183D MRI brain tumor segmentation using autoencoder regularizationcited by 324
  15. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  16. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  17. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  18. 2020Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classificationcited by 175
  19. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  20. 2019Privacy-preserving Federated Brain Tumour Segmentationcited by 169
  21. 2018Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentationcited by 148
  22. 2020Federated Visual Classification with Real-World Data Distributioncited by 126
  23. 2020Multi-site fMRI Analysis Using Privacy-preserving Federated Learning and Domain Adaptation: ABIDE Resultscited by 122
  24. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  25. 2020Distilled One-Shot Federated Learningcited by 108
  26. 2019Diversity with Cooperation: Ensemble Methods for Few-Shot Classificationcited by 70
  27. 2018Collaborative Learning for Deep Neural Networkscited by 69
  28. 2020XOR Mixup: Privacy-Preserving Data Augmentation for One-Shot Federated Learningcited by 58
  29. 2019Ensemble Knowledge Distillation for Learning Improved and Efficient Networkscited by 46
  30. 2020Weakly Supervised Lesion Localization With Probabilistic-CAM Poolingcited by 35
  31. 2016Learning Privately from Multiparty Datacited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 11; 2024: 13; 2025: 15; 2026: 3.

45 papers.

Sort:

By area:

  1. 2022A Survey on Heterogeneous Federated Learningcited by 42
  2. 2023Eliminating Domain Bias for Federated Learning in Representation Spacecited by 38
  3. 2023Federated Generative Learning with Foundation Modelscited by 23
  4. 2023A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiencycited by 21
  5. 2024Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchorscited by 21
  6. 2024Federated Distillation: A Surveycited by 14
  7. 2024Revisiting Ensembling in One-Shot Federated Learningcited by 13
  8. 2023Federated Learning over Coupled Graphscited by 12
  9. 2024Noiseless Privacy-Preserving Decentralized Learningcited by 9
  10. 2024A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applicationscited by 8
  11. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 8
  12. 2025dFLMoE: Decentralized Federated Learning via Mixture of Experts for Medical Data Analysiscited by 8
33 more
  1. 2024Direct Distillation between Different Domainscited by 7
  2. 2023Federated Learning via Input-Output Collaborative Distillationcited by 6
  3. 2025One-shot Federated Learning Methods: A Practical Guidecited by 6
  4. 2022Decentralized Learning with Multi-Headed Distillationcited by 5
  5. 2023Little is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learningcited by 5
  6. 2024On the Byzantine-Resilience of Distillation-Based Federated Learningcited by 5
  7. 2023FedPDD: A Privacy-preserving Double Distillation Framework for Cross-silo Federated Recommendationcited by 4
  8. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  9. 2023Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectivescited by 3
  10. 2024Linkage on Security, Privacy and Fairness in Federated Learning: New Balances and New Perspectivescited by 3
  11. 2025FedBEns: One-Shot Federated Learning based on Bayesian Ensemblecited by 3
  12. 2025Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practicecited by 3
  13. 2024Privacy-Preserving Federated Learning with Consistency via Knowledge Distillation Using Conditional Generatorcited by 2
  14. 2024DFDG: Data-Free Dual-Generator Adversarial Distillation for One-Shot Federated Learningcited by 2
  15. 2025Corrected with the Latest Version: Make Robust Asynchronous Federated Learning Possiblecited by 2
  16. 2025Robust Federated Inferencecited by 2
  17. 2022Domain Discrepancy Aware Distillation for Model Aggregation in Federated Learningcited by 1
  18. 2025Data-Free Black-Box Federated Learning via Zeroth-Order Gradient Estimationcited by 1
  19. 2025Soft-Label Caching and Sharpening for Communication-Efficient Federated Distillationcited by 1
  20. 2025FedSaaS: Class-Consistency Federated Semantic Segmentation via Global Prototype Supervision and Local Adversarial Harmonizationcited by 1
  21. 2025Adaptive Federated Distillation for Multi-Domain Non-IID Textual Datacited by 1
  22. 2025FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learningcited by 1
  23. 2026Distribution Alignment for One-Shot Federated Learning via Optimal Transportcited by 1
  24. 2023Model-Contrastive Federated Domain Adaptationcited by 0
  25. 2023Heterogeneous Federated Learning via Personalized Generative Networkscited by 0
  26. 2023REFT: Resource-Efficient Federated Training Framework for Heterogeneous and Resource-Constrained Environmentscited by 0
  27. 2024Harnessing Increased Client Participation with Cohort-Parallel Federated Learningcited by 0
  28. 2024FedTSA: A Cluster-based Two-Stage Aggregation Method for Model-heterogeneous Federated Learningcited by 0
  29. 2025Class-Imbalanced-Aware Adaptive Dataset Distillation for Scalable Pretrained Model on Credit Scoringcited by 0
  30. 2025ATR-Bench: A Federated Learning Benchmark for Adaptation, Trust, and Reasoningcited by 0
  31. 2025Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Datacited by 0
  32. 2026FedDAP: Domain-Aware Prototype Learning for Federated Learning under Domain Shiftcited by 0
  33. 2026Out-of-Distribution Federated Distillation with Domain-Aware Proxycited by 0