mlsanityData to 7 Oct 2026

Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation

arXiv 2204.07028 ↗First posted 14 Apr 2022cs.LG cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is a privacy-preserving machine learning paradigm in which the server periodically aggregates local model parameters from clients without assembling their private data. Constrained communication and personalization requirements pose severe challenges to FL. Federated distillation (FD) is proposed to simultaneously address the above two problems, which exchanges knowledge between the server and clients, supporting heterogeneous local models while significantly reducing communication overhead. However, most existing FD methods require a proxy dataset, which is often unavailable in reality. A few recent proxy-data-free FD approaches can eliminate the need for additional public data, but suffer from remarkable discrepancy among local knowledge due to client-side model heterogeneity, leading to ambiguous representation on the server and inevitable accuracy degradation. To tackle this issue, we propose a proxy-data-free FD algorithm based on distributed knowledge congruence (FedDKC). FedDKC leverages well-designed refinement strategies to narrow local knowledge differences into an acceptable upper bound, so as to mitigate the negative effects of knowledge incongruence. Specifically, from perspectives of peak probability and Shannon entropy of local knowledge, we design kernel-based knowledge refinement (KKR) and searching-based knowledge refinement (SKR) respectively, and theoretically guarantee that the refined-local knowledge can satisfy an approximately-similar distribution and be regarded as congruent. Extensive experiments conducted on three common datasets demonstrate that our proposed FedDKC significantly outperforms the state-of-the-art on various heterogeneous settings while evidently improving the convergence speed.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 27 of its 44 references that are in the map. First posted on arXiv 14 Apr 2022.

Builds on

27 of 44 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  7. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  8. 2020The Future of Digital Health with Federated Learningcited by 625
  9. 2019Federated Learning with Personalization Layerscited by 533
  10. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  11. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  12. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
15 more
  1. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  2. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 298
  3. 2019Correlation Congruence for Knowledge Distillationcited by 280
  4. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  5. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  6. 2020Federated Learning of a Mixture of Global and Local Modelscited by 208
  7. 2018Large scale distributed neural network training through online distillationcited by 144
  8. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  9. 2019Knowledge Adaptation for Efficient Semantic Segmentationcited by 105
  10. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 91
  11. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  12. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Datacited by 31
  13. 2021Spirit Distillation: A Model Compression Method with Multi-domain Knowledge Transfercited by 7
  14. 2018Custom Dual Transportation Mode Detection by Smartphone Devices Exploiting Sensor Diversitycited by 5
  15. 2021Spirit Distillation: Precise Real-time Semantic Segmentation of Road Scenes with Insufficient Datacited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 8; 2025: 5; 2026: 2.

22 papers.

Sort:

  1. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  2. 2023FedICT: Federated Multi-task Distillation for Multi-access Edge Computingcited by 17
  3. 2024Advances in Robust Federated Learning: A Survey with Heterogeneity Considerationscited by 12
  4. 2023Agglomerative Federated Learning: Empowering Larger Model Training via End-Edge-Cloud Collaborationcited by 10
  5. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 8
  6. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
  7. 2023Federated Learning on Virtual Heterogeneous Data with Local-global Distillationcited by 6
  8. 2023Knowledge Distillation in Federated Edge Learning: A Surveycited by 5
  9. 2023Federated Skewed Label Learning with Logits Fusioncited by 5
  10. 2024Federated Class-Incremental Learning with New-Class Augmented Self-Distillationcited by 4
  11. 2025FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Modelscited by 2
  12. 2023Improving Communication Efficiency of Federated Distillation via Accumulating Local Updatescited by 1
10 more
  1. 2024Logits Poisoning Attack in Federated Distillationcited by 1
  2. 2024FedCache 2.0: Federated Edge Learning with Knowledge Caching and Dataset Distillationcited by 1
  3. 2024Peak-Controlled Logits Poisoning Attack in Federated Distillationcited by 1
  4. 2025Beyond Model Scale Limits: End-Edge-Cloud Federated Learning with Self-Rectified Knowledge Agglomerationcited by 1
  5. 2025FedMHO: Heterogeneous One-Shot Federated Learning Towards Resource-Constrained Edge Devicescited by 1
  6. 2025Hypernetworks for Model-Heterogeneous Personalized Federated Learningcited by 1
  7. 2024Federated Knowledge Transfer Fine-tuning Large Server Model with Resource-Constrained IoT Clientscited by 0
  8. 2025HFedCKD: Toward Robust Heterogeneous Federated Learning via Data-free Knowledge Distillation and Two-way Contrastcited by 0
  9. 2026EFFEKT: Efficient Federated Knowledge Transfer to Foundation Modelscited by 0
  10. 2026FedADB: Class Anchor-Driven Dual-Branch Federated Learning for Mitigating Forgettingcited by 0