mlsanityData to 14 Oct 2026

FedDW: Distilling Weights through Consistency Optimization in Heterogeneous Federated Learning

arXiv 2412.04521 ↗First posted 5 Dec 2024cs.LG cs.CEcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) is an innovative distributed machine learning paradigm that enables neural network training across devices without centralizing data. While this addresses issues of information sharing and data privacy, challenges arise from data heterogeneity across clients and increasing network scale, leading to impacts on model performance and training efficiency. Previous research shows that in IID environments, the parameter structure of the model is expected to adhere to certain specific consistency principles. Thus, identifying and regularizing these consistencies can mitigate issues from heterogeneous data. We found that both soft labels derived from knowledge distillation and the classifier head parameter matrix, when multiplied by their own transpose, capture the intrinsic relationships between data classes. These shared relationships suggest inherent consistency. Therefore, the work in this paper identifies the consistency between the two and leverages it to regulate training, underpinning our proposed FedDW framework. Experimental results show FedDW outperforms 10 state-of-the-art FL methods, improving accuracy by an average of 3% in highly heterogeneous settings. Additionally, we provide a theoretical proof that FedDW offers higher efficiency, with the additional computational load from backpropagation being negligible. The code is available at https://github.com/liuvvvvv1/FedDW.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 45 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

14 of 45 references.

References · 14

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  4. 2018Federated Learning with Non-IID Datacited by 1,108
  5. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 809
  6. 2019Federated Learning with Personalization Layerscited by 541
  7. 2024Large Language Models: A Surveycited by 525
  8. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  9. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
  10. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 182
  11. 2024MH-pFLID: Model Heterogeneous personalized Federated Learning via Injection and Distillation for Medical Data Analysiscited by 14
  12. 2024Exploring Vacant Classes in Label-Skewed Federated Learningcited by 10
2 more
  1. 2023CLIP-guided Federated Learning on Heterogeneous and Long-Tailed Datacited by 9
  2. 2024CPCL: Cross-Modal Prototypical Contrastive Learning for Weakly Supervised Text-based Person Retrievalcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Adaptive Self-Distillation for Minimizing Client Drift in Heterogeneous Federated Learning
  2. 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning
  3. 2024Training Heterogeneous Client Models using Knowledge Distillation in Serverless Federated Learning
  4. 2022Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
  5. 2021FedHe: Heterogeneous Models and Communication-Efficient Federated Learning
  6. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning
  7. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning
  8. 2025Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning