FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated Learning
Not on the map. Why?
Abstract (arXiv)
In recent years, federated learning (FL) has emerged as a promising technique for training machine learning models in a decentralized manner while also preserving data privacy. The non-independent and identically distributed (non-i.i.d.) nature of client data, coupled with constraints on client or edge devices, presents significant challenges in FL. Furthermore, learning across a high number of communication rounds can be risky and potentially unsafe for model exploitation. Traditional FL approaches may suffer from these challenges. Therefore, we introduce FedSiKD, which incorporates knowledge distillation (KD) within a similarity-based federated learning framework. As clients join the system, they securely share relevant statistics about their data distribution, promoting intra-cluster homogeneity. This enhances optimization efficiency and accelerates the learning process, effectively transferring knowledge between teacher and student models and addressing device constraints. FedSiKD outperforms state-of-the-art algorithms by achieving higher accuracy, exceeding by 25% and 18% for highly skewed data at α = 0.1,0.5 on the HAR and MNIST datasets, respectively. Its faster convergence is illustrated by a 17% and 20% increase in accuracy within the first five rounds on the HAR and MNIST datasets, respectively, highlighting its early-stage learning proficiency. Code is publicly available and hosted on GitHub (https://github.com/SimuEnv/FedSiKD)
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 46 references that are in the map. First posted on arXiv 14 Feb 2024.
Builds on
23 of 46 references.
References · 23
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2018Federated Learning with Non-IID Datacited by 1,099
- 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
- 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
- 2021Federated Learning on Non-IID Data: A Surveycited by 345
- 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
- 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
- 2020Federated learning with hierarchical clustering of local updates to improve training on non-IID datacited by 222
- 2020Federated Visual Classification with Real-World Data Distributioncited by 126
- 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 108
- 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
11 more
- 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
- 2021Node Selection Toward Faster Convergence for Federated Learning on Non-IID Datacited by 39
- 2019Hybrid-FL for Wireless Networks: Cooperative Learning Mechanism Using Non-IID Datacited by 38
- 2022MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized Healthcarecited by 36
- 2022Faster Adaptive Federated Learningcited by 35
- 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Datacited by 31
- 2022Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matchingcited by 24
- 2022FLIS: Clustered Federated Learning via Inference Similarity for Non-IID Data Distributioncited by 22
- 2020FedFMC: Sequential Efficient Federated Learning on Non-iid Datacited by 17
- 2023FedICT: Federated Multi-task Distillation for Multi-access Edge Computingcited by 17
- 2023Resource Aware Clustering for Tackling the Heterogeneity of Participants in Federated Learningcited by 2
Built on by
Not cited yet.