Domain Discrepancy Aware Distillation for Model Aggregation in Federated Learning
Not on the map. Why?
Abstract (arXiv)
Knowledge distillation has recently become popular as a method of model aggregation on the server for federated learning. It is generally assumed that there are abundant public unlabeled data on the server. However, in reality, there exists a domain discrepancy between the datasets of the server domain and a client domain, which limits the performance of knowledge distillation. How to improve the aggregation under such a domain discrepancy setting is still an open problem. In this paper, we first analyze the generalization bound of the aggregation model produced from knowledge distillation for the client domains, and then describe two challenges, server-to-client discrepancy and client-to-client discrepancy, brought to the aggregation model by the domain discrepancies. Following our analysis, we propose an adaptive knowledge aggregation algorithm FedD3A based on domain discrepancy aware distillation to lower the bound. FedD3A performs adaptive weighting at the sample level in each round of FL. For each sample in the server domain, only the client models of its similar domains will be selected for playing the teacher role. To achieve this, we show that the discrepancy between the server-side sample and the client domain can be approximately measured using a subspace projection matrix calculated on each client without accessing its raw data. The server can thus leverage the projection matrices from multiple clients to assign weights to the corresponding teacher models for each server-side sample. We validate FedD3A on two popular cross-domain datasets and show that it outperforms the compared competitors in both cross-silo and cross-device FL settings.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 4 Oct 2022.
Builds on
38 of 50 references.
References · 38
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2019Searching for MobileNetV3cited by 1,984
- 2014FitNets: Hints for Thin Deep Netscited by 1,619
- 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
- 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
- 2020Adaptive Federated Optimizationcited by 1,036
- 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
26 more
- 2019Relational Knowledge Distillationcited by 873
- 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
- 2020Personalized Federated Learning with Moreau Envelopescited by 612
- 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
- 2020Federated Learning with Matched Averagingcited by 564
- 2019Similarity-Preserving Knowledge Distillationcited by 507
- 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
- 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
- 2021See through Gradients: Image Batch Recovery via GradInversioncited by 320
- 2019Model Fusion via Optimal Transportcited by 217
- 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
- 2018Continual Learning of Context-dependent Processing in Neural Networkscited by 178
- 2017Learning from Noisy Labels with Distillationcited by 160
- 2019Federated Adversarial Domain Adaptationcited by 140
- 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
- 2019One-Shot Federated Learningcited by 124
- 2020Lower Bounds and Optimal Algorithms for Personalized Federated Learningcited by 122
- 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
- 2019Multi-source Distilling Domain Adaptationcited by 73
- 2020Federated Learning with Only Positive Labelscited by 55
- 2018MEAL: Multi-Model Ensemble via Adversarial Learningcited by 52
- 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillationcited by 45
- 2020Federated Model Distillation with Noise-Free Differential Privacycited by 40
- 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Datacited by 31
- 2022One-shot Federated Learning without Server-side Trainingcited by 20
- 2021FedRAD: Federated Robust Adaptive Distillationcited by 11
Built on by
Citing papers per year: 2024: 1.
1 paper.