mlsanityData to 14 Oct 2026

DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning

arXiv 2309.13546 ↗First posted 24 Sep 2023cs.CV cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) is a privacy-constrained decentralized machine learning paradigm in which clients enable collaborative training without compromising private data. However, how to learn a robust global model in the data-heterogeneous and model-heterogeneous FL scenarios is challenging. To address it, we resort to data-free knowledge distillation to propose a new FL method (namely DFRD). DFRD equips a conditional generator on the server to approximate the training space of the local models uploaded by clients, and systematically investigates its training in terms of fidelity, transferability} and diversity. To overcome the catastrophic forgetting of the global model caused by the distribution shifts of the generator across communication rounds, we maintain an exponential moving average copy of the generator on the server. Additionally, we propose dynamic weighting and label sampling to accurately extract knowledge from local models. Finally, our extensive experiments on various image classification tasks illustrate that DFRD achieves significant performance gains compared to SOTA baselines.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 61 of its 81 references that are in the map. First posted on arXiv 24 Sep 2023.

Builds on

61 of 81 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2019Advances and Open Problems in Federated Learningcited by 3,046
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  7. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  8. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  9. 2018Federated Learning with Non-IID Datacited by 1,108
  10. 2021Model-Contrastive Federated Learningcited by 766
  11. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  12. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
38 more
  1. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  2. 2016Conditional Image Synthesis With Auxiliary Classifier GANscited by 565
  3. 2020Federated Learning with Matched Averagingcited by 564
  4. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  5. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 459
  6. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  7. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  8. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 408
  9. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  10. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  11. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  12. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  13. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  14. 2021Federated Learning for Internet of Things: A Comprehensive Surveycited by 252
  15. 2019Data-Free Learning of Student Networkscited by 204
  16. 2015Federated Optimization:Distributed Optimization Beyond the Datacentercited by 195
  17. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  18. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  19. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 163
  20. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  21. 2019Zero-Shot Knowledge Distillation in Deep Networkscited by 121
  22. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  23. 2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 109
  24. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  25. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  26. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  27. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  28. 2021Federated Reconstruction: Partially Local Federated Learningcited by 79
  29. 2018AI Benchmark: Running Deep Neural Networks on Android Smartphonescited by 75
  30. 2021Contrastive Model Inversion for Data-Free Knowledge Distillationcited by 66
  31. 2022Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learningcited by 52
  32. 2021Privacy-preserving Federated Learning based on Multi-key Homomorphic Encryptioncited by 48
  33. 2022Towards Practical Certifiable Patch Defense with Vision Transformercited by 44
  34. 2022No One Left Behind: Inclusive Federated Learning over Heterogeneous Devicescited by 37
  35. 2022Differentially Private Federated Learning with Local Regularization and Sparsificationcited by 36
  36. 2022Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replaycited by 35
  37. 2022Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customizationcited by 35
  38. 2022Multi-Level Branched Regularization for Federated Learningcited by 34

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 11; 2026: 2.

24 papers.

Sort:

By area:

  1. 2024Text-Enhanced Data-free Approach for Federated Class-Incremental Learningcited by 16
  2. 2024Federated Distillation: A Surveycited by 15
  3. 2024Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directionscited by 13
  4. 2024Exploring Vacant Classes in Label-Skewed Federated Learningcited by 10
  5. 2023FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learningcited by 5
  6. 2025FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Datacited by 5
  7. 2025Federated Deconfounding and Debiasing Learning for Out-of-Distribution Generalizationcited by 4
  8. 2024FedHPL: Efficient Heterogeneous Federated Learning with Prompt Tuning and Logit Distillationcited by 3
  9. 2024FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learningcited by 2
  10. 2024Privacy-Preserving Federated Learning with Consistency via Knowledge Distillation Using Conditional Generatorcited by 2
  11. 2024DFDG: Data-Free Dual-Generator Adversarial Distillation for One-Shot Federated Learningcited by 2
  12. 2024Federated Hybrid Training and Self-Adversarial Distillation: Towards Robust Edge Networkscited by 2
12 more
  1. 2025FedBM: Stealing Knowledge from Pre-trained Language Models for Heterogeneous Federated Learningcited by 2
  2. 2025Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environmentscited by 2
  3. 2025Towards Communication-Efficient Adversarial Federated Learning for Robust Edge Intelligencecited by 1
  4. 2025FedMHO: Heterogeneous One-Shot Federated Learning Towards Resource-Constrained Edge Devicescited by 1
  5. 2025Data-Free Black-Box Federated Learning via Zeroth-Order Gradient Estimationcited by 1
  6. 2025Enhancing Federated Class-Incremental Learning via Spatial-Temporal Statistics Aggregationcited by 1
  7. 2025Hypernetworks for Model-Heterogeneous Personalized Federated Learningcited by 1
  8. 2024Adaptive Guidance for Local Training in Heterogeneous Federated Learningcited by 0
  9. 2025HFedCKD: Toward Robust Heterogeneous Federated Learning via Data-free Knowledge Distillation and Two-way Contrastcited by 0
  10. 2025DART: A Server-side Plug-in for Resource-efficient Robust Federated Learningcited by 0
  11. 2026From Data Heterogeneity to Convergence: A Data-Centric Review of Federated Learningcited by 0
  12. 2026Distributed Subliminal Learning: Replacing Model Updates with Random-Carrier Outputscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024FedDW: Distilling Weights through Consistency Optimization in Heterogeneous Federated Learning
  2. 2023Heterogeneous Federated Learning Using Knowledge Codistillation
  3. 2022Knowledge Distillation for Federated Learning: a Practical Guide
  4. 2024Training Heterogeneous Client Models using Knowledge Distillation in Serverless Federated Learning
  5. 2022FedDKD: Federated Learning with Decentralized Knowledge Distillation
  6. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions
  7. 2025FedBKD: Distilled Federated Learning to Embrace Gerneralization and Personalization on Non-IID Data
  8. 2025Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data