mlsanityData to 14 Oct 2026

Decentralized Learning with Multi-Headed Distillation

arXiv 2211.15774 ↗First posted 28 Nov 2022cs.LG cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Decentralized learning with private data is a central problem in machine learning. We propose a novel distillation-based decentralized learning technique that allows multiple agents with private non-iid data to learn from each other, without having to share their data, weights or weight updates. Our approach is communication efficient, utilizes an unlabeled public dataset and uses multiple auxiliary heads for each client, greatly improving training efficiency in the case of heterogeneous data. This approach allows individual models to preserve and enhance performance on their private tasks while also dramatically improving their performance on the global aggregated data distribution. We study the effects of data and model architecture heterogeneity and the impact of the underlying communication graph topology on learning efficiency and show that our agents can significantly improve their performance compared to learning in isolation.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 34 of its 47 references that are in the map. First posted on arXiv 28 Nov 2022.

Builds on

34 of 47 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  6. 2018Born Again Neural Networkscited by 546
  7. 2013Do Deep Nets Really Need to be Deep?cited by 474
  8. 2021Towards Personalized Federated Learningcited by 464
  9. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  10. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
  11. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  12. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
22 more
  1. 2019Variational Information Distillation for Knowledge Transfercited by 324
  2. 2018Dataset Distillationcited by 230
  3. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
  4. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 187
  5. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  6. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 173
  7. 2020Survey of Personalization Techniques for Federated Learningcited by 145
  8. 2019One-Shot Federated Learningcited by 125
  9. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  10. 2022Federated Learning with Partial Model Personalizationcited by 114
  11. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 109
  12. 2019Knowledge Distillation from Internal Representationscited by 87
  13. 2020Federated Mutual Learningcited by 71
  14. 2018Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Studentscited by 50
  15. 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillationcited by 45
  16. 2020Federated Model Distillation with Noise-Free Differential Privacycited by 40
  17. 2022Architecture Agnostic Federated Learning for Neural Networkscited by 30
  18. 2020Communication-Efficient Federated Distillationcited by 28
  19. 2021FedRAD: Federated Robust Adaptive Distillationcited by 11
  20. 2022Personalized Federated Learning via Heterogeneous Modular Networkscited by 8
  21. 2022CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learningcited by 6
  22. 2020Adaptive Distillation for Decentralized Learning from Heterogeneous Clientscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 1.

5 papers.

  1. 2024Dataset Distillation by Automatic Training Trajectoriescited by 26
  2. 2024DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Modelscited by 3
  3. 2025On the Fast Adaptation of Delayed Clients in Decentralized Federated Learning: A Centroid-Aligned Distillation Approachcited by 1
  4. 2026CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learningcited by 1
  5. 2024Harnessing Increased Client Participation with Cohort-Parallel Federated Learningcited by 0