mlsanityData to 7 Oct 2026

Data-Free Knowledge Distillation for Heterogeneous Federated Learning

arXiv 2105.10056 ↗First posted 20 May 2021cs.LG cs.DCcited by 380

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) is a decentralized machine-learning paradigm, in which a global server iteratively averages the model parameters of local users without accessing their data. User heterogeneity has imposed significant challenges to FL, which can incur drifted global models that are slow to converge. Knowledge Distillation has recently emerged to tackle this issue, by refining the server model using aggregated knowledge from heterogeneous users, other than directly averaging their model parameters. This approach, however, depends on a proxy dataset, making it impractical unless such a prerequisite is satisfied. Moreover, the ensemble knowledge is not fully utilized to guide local model learning, which may in turn affect the quality of the aggregated model. Inspired by the prior art, we propose a data-free knowledge distillation} approach to address heterogeneous FL, where the server learns a lightweight generator to ensemble user information in a data-free manner, which is then broadcasted to users, regulating local training using the learned knowledge as an inductive bias. Empirical studies powered by theoretical implications show that, our approach facilitates FL with better generalization performance using fewer communication rounds, compared with the state-of-the-art.

Key facts. Cited by 380 papers in the map (data through 7 Oct 2026). Builds on 37 of its 56 references that are in the map. First posted on arXiv 20 May 2021.

Builds on

37 of 56 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2013Auto-Encoding Variational Bayescited by 3,980
  6. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  7. 2019Advances and Open Problems in Federated Learningcited by 3,021
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  9. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  10. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  11. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  12. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
25 more
  1. 2018LEAF: A Benchmark for Federated Settingscited by 780
  2. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  3. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  4. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  5. 2020Federated Learning with Matched Averagingcited by 564
  6. 2013Do Deep Nets Really Need to be Deep?cited by 466
  7. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  8. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  9. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
  10. 2020Attack of the Tails: Yes, You Really Can Backdoor Federated Learningcited by 307
  11. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  12. 2018Dataset Distillationcited by 224
  13. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  14. 2018cpSGD: Communication-efficient and differentially-private distributed SGDcited by 157
  15. 2019Federated Adversarial Domain Adaptationcited by 140
  16. 2019Mode Seeking Generative Adversarial Networks for Diverse Image Synthesiscited by 138
  17. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  18. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  19. 2019One-Shot Federated Learningcited by 124
  20. 2019Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation Systemcited by 79
  21. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 67
  22. 2020Federated Knowledge Distillationcited by 51
  23. 2020Federated Model Distillation with Noise-Free Differential Privacycited by 40
  24. 2021FedFace: Collaborative Learning of Face Recognition Modelcited by 26
  25. 2019On Convergence of Distributed Approximate Newton Methods: Globalization, Sharper Bounds and Beyondcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 59; 2023: 105; 2024: 81; 2025: 84; 2026: 35.

380 papers.

Sort:

By area:

  1. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  2. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  3. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
  4. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  5. 2022FedALA: Adaptive Local Aggregation for Personalized Federated Learningcited by 121
  6. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  7. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  8. 2023Deep Model Fusion: A Surveycited by 89
  9. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  10. 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 66
  11. 2021Personalized Federated Learning through Local Memorizationcited by 57
  12. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 57
38 more
  1. 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
  2. 2022Personalized Subgraph Federated Learningcited by 54
  3. 2023Better Generative Replay for Continual Federated Learningcited by 53
  4. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
  5. 2024FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learningcited by 47
  6. 2022ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsitycited by 45
  7. 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillationcited by 45
  8. 2022pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learningcited by 44
  9. 2022A Survey on Heterogeneous Federated Learningcited by 42
  10. 2023A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Taskscited by 41
  11. 2021FedCG: Leverage Conditional GAN for Protecting Privacy and Maintaining Competitive Performance in Federated Learningcited by 39
  12. 2022Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imagingcited by 39
  13. 2023Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learningcited by 39
  14. 2023Efficient Personalized Federated Learning via Sparse Model-Adaptationcited by 38
  15. 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 38
  16. 2023Eliminating Domain Bias for Federated Learning in Representation Spacecited by 38
  17. 2023Personalized Federated Learning under Mixture of Distributionscited by 37
  18. 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
  19. 2022Differentially Private Federated Learning with Local Regularization and Sparsificationcited by 35
  20. 2022Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customizationcited by 34
  21. 2022Multi-Level Branched Regularization for Federated Learningcited by 34
  22. 2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformerscited by 33
  23. 2023FedNoRo: Towards Noise-Robust Federated Learning by Addressing Class Imbalance and Label Noise Heterogeneitycited by 33
  24. 2024Towards Efficient Replay in Federated Incremental Learningcited by 33
  25. 2024An Aggregation-Free Federated Learning for Tackling Data Heterogeneitycited by 33
  26. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Datacited by 31
  27. 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
  28. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  29. 2021Data-Free Knowledge Transfer: A Surveycited by 31
  30. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  31. 2022Disentangled Federated Learning for Tackling Attributes Skew via Invariant Aggregation and Diversity Transferringcited by 28
  32. 2022FedFM: Anchor-based Feature Matching for Data Heterogeneity in Federated Learningcited by 27
  33. 2023GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgettingcited by 27
  34. 2023Understanding Generalization of Federated Learning via Stability: Heterogeneity Matterscited by 27
  35. 2024Enhancing One-Shot Federated Learning Through Data and Ensemble Co-Boostingcited by 27
  36. 2024FedTAD: Topology-aware Data-free Knowledge Distillation for Subgraph Federated Learningcited by 27
  37. 2021Personalized Federated Learning with Adaptive Batchnorm for Healthcarecited by 26
  38. 2022Meta Knowledge Condensation for Federated Learningcited by 26

50 most cited of 380 All 380 (JSON)