# Adaptive Distillation for Decentralized Learning from Heterogeneous Clients

arXiv: https://arxiv.org/abs/2008.07948 · HTML: https://mlsanity.com/paper/2008.07948/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Aug 2020.

## Built on by (most cited first, 5 of 5)

- [Decentralized Federated Learning through Proxy Model Sharing](https://mlsanity.com/paper/2111.11343/) (2021)
- [Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer](https://mlsanity.com/paper/2109.08119/) (2021)
- [Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions](https://mlsanity.com/paper/2406.10861/) (2024)
- [Decentralized Learning with Multi-Headed Distillation](https://mlsanity.com/paper/2211.15774/) (2022)
- [Adaptive Dual-Mode Distillation with Incentive Schemes for Scalable, Heterogeneous Federated Learning on Non-IID Data](https://mlsanity.com/paper/2509.22507/) (2025)

## Abstract (arXiv)

This paper addresses the problem of decentralized learning to achieve a high-performance global model by asking a group of clients to share local models pre-trained with their own data resources. We are particularly interested in a specific case where both the client model architectures and data distributions are diverse, which makes it nontrivial to adopt conventional approaches such as Federated Learning and network co-distillation. To this end, we propose a new decentralized learning method called Decentralized Learning via Adaptive Distillation (DLAD). Given a collection of client models and a large number of unlabeled distillation samples, the proposed DLAD 1) aggregates the outputs of the client models while adaptively emphasizing those with higher confidence in given distillation samples and 2) trains the global model to imitate the aggregated outputs. Our extensive experimental evaluation on multiple public datasets (MNIST, CIFAR-10, and CINIC-10) demonstrates the effectiveness of the proposed method.