# FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated Learning

arXiv: https://arxiv.org/abs/2402.09095 · HTML: https://mlsanity.com/paper/2402.09095/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 14 Feb 2024.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 23 of 23)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data](https://mlsanity.com/paper/1610.05755/) (2016)
- [Federated Learning on Non-IID Data: A Survey](https://mlsanity.com/paper/2106.06843/) (2021)
- [Three Approaches for Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2002.10619/) (2020)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Federated learning with hierarchical clustering of local updates to improve training on non-IID data](https://mlsanity.com/paper/2004.11791/) (2020)
- [Federated Visual Classification with Real-World Data Distribution](https://mlsanity.com/paper/2003.08082/) (2020)
- [Parameterized Knowledge Transfer for Personalized Federated Learning](https://mlsanity.com/paper/2111.02862/) (2021)
- [Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning](https://mlsanity.com/paper/2204.12703/) (2022)
- [FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion](https://mlsanity.com/paper/2110.11027/) (2021)
- [Node Selection Toward Faster Convergence for Federated Learning on Non-IID Data](https://mlsanity.com/paper/2105.07066/) (2021)
- [Hybrid-FL for Wireless Networks: Cooperative Learning Mechanism Using Non-IID Data](https://mlsanity.com/paper/1905.07210/) (2019)
- [MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized Healthcare](https://mlsanity.com/paper/2206.08516/) (2022)
- [Faster Adaptive Federated Learning](https://mlsanity.com/paper/2212.00974/) (2022)
- [Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data](https://mlsanity.com/paper/2107.00051/) (2021)
- [Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matching](https://mlsanity.com/paper/2203.12285/) (2022)
- [FLIS: Clustered Federated Learning via Inference Similarity for Non-IID Data Distribution](https://mlsanity.com/paper/2208.09754/) (2022)
- [FedFMC: Sequential Efficient Federated Learning on Non-iid Data](https://mlsanity.com/paper/2006.10937/) (2020)
- [FedICT: Federated Multi-task Distillation for Multi-access Edge Computing](https://mlsanity.com/paper/2301.00389/) (2023)
- [Resource Aware Clustering for Tackling the Heterogeneity of Participants in Federated Learning](https://mlsanity.com/paper/2306.04207/) (2023)

## Abstract (arXiv)

In recent years, federated learning (FL) has emerged as a promising technique for training machine learning models in a decentralized manner while also preserving data privacy. The non-independent and identically distributed (non-i.i.d.) nature of client data, coupled with constraints on client or edge devices, presents significant challenges in FL. Furthermore, learning across a high number of communication rounds can be risky and potentially unsafe for model exploitation. Traditional FL approaches may suffer from these challenges. Therefore, we introduce FedSiKD, which incorporates knowledge distillation (KD) within a similarity-based federated learning framework. As clients join the system, they securely share relevant statistics about their data distribution, promoting intra-cluster homogeneity. This enhances optimization efficiency and accelerates the learning process, effectively transferring knowledge between teacher and student models and addressing device constraints. FedSiKD outperforms state-of-the-art algorithms by achieving higher accuracy, exceeding by 25% and 18% for highly skewed data at α = 0.1,0.5 on the HAR and MNIST datasets, respectively. Its faster convergence is illustrated by a 17% and 20% increase in accuracy within the first five rounds on the HAR and MNIST datasets, respectively, highlighting its early-stage learning proficiency. Code is publicly available and hosted on GitHub (https://github.com/SimuEnv/FedSiKD)