# Decentralized Federated Learning via Mutual Knowledge Transfer

arXiv: https://arxiv.org/abs/2012.13063 · HTML: https://mlsanity.com/paper/2012.13063/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Dec 2020.

## Built on by (most cited first, 30 of 43)

- [Federated Learning on Non-IID Data: A Survey](https://mlsanity.com/paper/2106.06843/) (2021)
- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges](https://mlsanity.com/paper/2211.08413/) (2022)
- [Decentralized Federated Learning through Proxy Model Sharing](https://mlsanity.com/paper/2111.11343/) (2021)
- [Deep Federated Learning for Autonomous Driving](https://mlsanity.com/paper/2110.05754/) (2021)
- [A Survey on Heterogeneous Federated Learning](https://mlsanity.com/paper/2210.04505/) (2022)
- [A Survey on Decentralized Federated Learning](https://mlsanity.com/paper/2308.04604/) (2023)
- [Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matching](https://mlsanity.com/paper/2203.12285/) (2022)
- [Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchors](https://mlsanity.com/paper/2405.11525/) (2024)
- [Find Your Friends: Personalized Federated Learning with the Right Collaborators](https://mlsanity.com/paper/2210.06597/) (2022)
- [Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions](https://mlsanity.com/paper/2406.10861/) (2024)
- [Advancements in Federated Learning: Models, Methods, and Privacy](https://mlsanity.com/paper/2302.11466/) (2023)
- [Knowledge Distillation in Federated Edge Learning: A Survey](https://mlsanity.com/paper/2301.05849/) (2023)
- [PEPPER: Empowering User-Centric Recommender Systems over Gossip Learning](https://mlsanity.com/paper/2208.05320/) (2022)
- [DFML: Decentralized Federated Mutual Learning](https://mlsanity.com/paper/2402.01863/) (2024)
- [DACFL: Dynamic Average Consensus Based Federated Learning in Decentralized Topology](https://mlsanity.com/paper/2111.05505/) (2021)
- [Homogenizing Non-IID datasets via In-Distribution Knowledge Distillation for Decentralized Learning](https://mlsanity.com/paper/2304.04326/) (2023)
- [Sparse Decentralized Federated Learning](https://mlsanity.com/paper/2308.16671/) (2023)
- [Federated Learning with Neural Graphical Models](https://mlsanity.com/paper/2309.11680/) (2023)
- [Practical Insights into Knowledge Distillation for Pre-Trained Models](https://mlsanity.com/paper/2402.14922/) (2024)
- [PDSL: Privacy-Preserved Decentralized Stochastic Learning with Heterogeneous Data Distribution](https://mlsanity.com/paper/2503.23726/) (2025)
- [Nonparametric Decentralized Detection and Sparse Sensor Selection via Multi-Sensor Online Kernel Scalar Quantization](https://mlsanity.com/paper/2205.10681/) (2022)
- [Reducing Training Time in Cross-Silo Federated Learning using Multigraph Topology](https://mlsanity.com/paper/2207.09657/) (2022)
- [Collaborative Distributed Machine Learning](https://mlsanity.com/paper/2309.16584/) (2023)
- [A review on different techniques used to combat the non-IID and heterogeneous nature of data in FL](https://mlsanity.com/paper/2401.00809/) (2024)
- [Gradient Coding in Decentralized Learning for Evading Stragglers](https://mlsanity.com/paper/2402.04193/) (2024)
- [FedSR: A Semi-Decentralized Federated Learning Algorithm for Non-IIDness in IoT System](https://mlsanity.com/paper/2403.14718/) (2024)
- [Adaptive Coded Federated Learning: Privacy Preservation and Straggler Mitigation](https://mlsanity.com/paper/2403.14905/) (2024)
- [Exposing the Vulnerability of Decentralized Learning to Membership Inference Attacks Through the Lens of Graph Mixing](https://mlsanity.com/paper/2412.12837/) (2024)
- [Federated prediction for scalable and privacy-preserved knowledge-based planning in radiotherapy](https://mlsanity.com/paper/2505.14507/) (2025)

## Abstract (arXiv)

In this paper, we investigate the problem of decentralized federated learning (DFL) in Internet of things (IoT) systems, where a number of IoT clients train models collectively for a common task without sharing their private training data in the absence of a central server. Most of the existing DFL schemes are composed of two alternating steps, i.e., model updating and model averaging. However, averaging model parameters directly to fuse different models at the local clients suffers from client-drift especially when the training data are heterogeneous across different clients. This leads to slow convergence and degraded learning performance. As a possible solution, we propose the decentralized federated earning via mutual knowledge transfer (Def-KT) algorithm where local clients fuse models by transferring their learnt knowledge to each other. Our experiments on the MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets reveal that the proposed Def-KT algorithm significantly outperforms the baseline DFL methods with model averaging, i.e., Combo and FullAvg, especially when the training data are not independent and identically distributed (non-IID) across different clients.